Guaranteed file system hierarchy data integrity in cloud object stores

ABSTRACT

Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional of and claims the benefit andpriority under 35 U.S.C. 119(e) of U.S. Provisional Application No.62/443,391, filed Jan. 6, 2017, entitled “FILE SYSTEM HIERARCHIES ANDFUNCTIONALITY WITH CLOUD OBJECT STORAGE,” the entire contents of whichare incorporated herein by reference for all purposes.

TECHNICAL FIELD OF THE INVENTION

This disclosure generally relates to systems and methods of datastorage, and more particularly to layering file system functionality onan object interface.

BACKGROUND

The continuous expansion of the Internet, along with the expansion andsophistication of computing networks and systems, has led to theproliferation of content being stored and accessible over the Internet.This, in turn, has driven the need for large and sophisticated datastorage systems. As the demand for data storage continues to increase,larger and more sophisticated storage systems are being designed anddeployed. Many large-scale data storage systems utilize storageappliances that include arrays of physical storage media. These storageappliances are capable of storing incredible amounts of data. Forexample, at this time, Oracle's SUN ZFS Storage ZS5-4 appliance canstore up to 6.9 petabytes of data. Moreover, multiple storage appliancesmay be networked together to form a storage pool, which can furtherincrease the volume of stored data.

Typically, large storage systems such as these may include a file systemfor storing and accessing files. In addition to storing system files(operating system files, device driver files, etc.), the file systemprovides storage and access of user data files. If any of these files(system files and/or user files) contain critical data, then it becomesadvantageous to employ a backup storage scheme to ensure that criticaldata is not lost if a file storage device fails.

Conventional cloud-based storage is object-based and offers elasticityand scale. However, cloud object storage presents a number of problems.Cloud object storage offers interfaces that are based on getting andputting whole objects. Cloud object storage provides a limited abilityto search, and typically has high latency. The limited cloud-basedinterfaces do not align with needs of local file system applications.Converting legacy applications to use an object interface would beexpensive and may not be practical or even possible. Cloud objectstorage encryption keeps encryption keys making data more vulnerable andless secure.

Thus, there is a need for systems and methods that address the foregoingproblems in order to provide layering of file system functionality on anobject interface. This and other needs are addressed by the presentdisclosure.

BRIEF SUMMARY

Certain embodiments of the present disclosure relate generally tosystems and methods of data storage, and more particularly to systemsand methods for layering file system functionality on an objectinterface.

Various techniques (e.g., systems, methods, computer-program productstangibly embodied in a non-transitory machine-readable storage medium,etc.) are described herein for providing layering of file systemfunctionality on an object interface. In certain embodiments, filesystem functionality may be layered on cloud object interfaces toprovide cloud-based storage while allowing for functionality expectedfrom a legacy applications. For instance, POSIX interfaces and semanticsmay be layered on cloud-based storage, while providing access to data ina manner consistent with file-based access with data organization inname hierarchies. Various embodiments also may provide for memorymapping of data so that memory map changes are reflected in persistentstorage while ensuring consistency between memory map changes andwrites. For example, by transforming a ZFS file system disk-basedstorage into ZFS cloud-based storage, the ZFS file system gains theelastic nature of cloud storage.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating various embodiments, are intended for purposes ofillustration only and are not intended to necessarily limit the scope ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of embodimentsaccording to the present disclosure may be realized by reference to theremaining portions of the specification in conjunction with thefollowing appended figures.

FIG. 1 illustrates one example storage network that may be used inaccordance with certain embodiments of the present disclosure.

FIG. 2 illustrates an instance of a file system that may be executed ina storage environment, in accordance with certain embodiments of thepresent disclosure.

FIGS. 3A-3F illustrate a copy-on-write process for a file system, inaccordance with certain embodiments of the present disclosure.

FIG. 4 is a high-level diagram illustrating an example of a hybrid cloudstorage system, in accordance with certain embodiments of the presentdisclosure.

FIG. 5 illustrates an instance of an example network file system of thehybrid cloud storage system, in accordance with certain embodiments ofthe present disclosure.

FIG. 6 is a diagram illustrating additional aspects of a cloud interfaceappliance of a hybrid cloud storage system, in accordance with certainembodiments of the present disclosure.

FIGS. 7A-F are block diagrams that illustrates an example methoddirected to certain features of a COW process for the hybrid cloudstorage system, in accordance with certain embodiments of the presentdisclosure including data services, snapshots, and clones.

FIG. 8 is a high-level diagram illustrating an example of the cloudinterface appliance handling incremental modifications, in accordancewith certain embodiments of the present disclosure.

FIG. 9 is a block diagram that illustrates an example method directed tocertain features of the hybrid cloud storage system that ensureintegrity in the cloud and always-consistent semantics from aneventually consistent object model, in accordance with certainembodiments of the present disclosure.

FIG. 10 is a high-level diagram illustrating an example of the cloudinterface appliance handling the checking, in accordance with certainembodiments of the present disclosure.

FIG. 11 is a diagram of a simplified example further illustratingfeatures of a hybrid cloud storage system, in accordance with certainembodiments of the present disclosure.

FIG. 12 is a diagram of a simplified example further illustratingfeatures of a hybrid cloud storage system, in accordance with certainembodiments of the present disclosure.

FIG. 13 is a block diagram that illustrates an example method directedto certain features of the hybrid cloud storage system for cachemanagement and cloud latency masking, in accordance with certainembodiments of the present disclosure.

FIG. 14 illustrates an instance of an example network file system of thehybrid cloud storage system to facilitate synchronous mirroring, inaccordance with certain embodiments of the present disclosure.

FIG. 15 is a block diagram that illustrates an example method directedto certain features of the hybrid cloud storage system for synchronousmirroring and cloud latency masking, in accordance with certainembodiments of the present disclosure.

FIG. 16 depicts a simplified diagram of a distributed system forimplementing certain embodiments in accordance with present disclosure.

FIG. 17 is a simplified block diagram of one or more components of asystem environment by which services provided by one or more componentsof a system may be offered as cloud services, in accordance with certainembodiments of the present disclosure.

FIG. 18 illustrates an exemplary computer system, in which variousembodiments of the present invention may be implemented.

In the appended figures, similar components and/or features may have thesame reference label. Further, various components of the same type maybe distinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION

The ensuing description provides preferred exemplary embodiment(s) only,and is not intended to limit the scope, applicability, or configurationof the disclosure. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodimentof the disclosure. It should be understood that various changes may bemade in the function and arrangement of elements without departing fromthe spirit and scope of the disclosure as set forth in the appendedclaims.

As noted above, cloud-based storage offers elasticity and scale, butpresents a number of problems. Cloud object storage offers interfacesthat are based on getting and putting whole objects. Cloud objectstorage provides a limited ability to search, and typically has highlatency. The limited cloud-based interfaces do not align with needs oflocal file system applications. Converting legacy applications to use anobject interface would be expensive and may not be practical or evenpossible. Hence, solutions are needed so that is not necessary to changefile system applications to directly access cloud object storage becauseit promises to be complex and expensive.

The solutions should allow native application interfaces to be preservedwithout introducing various types of adaptation layers to map the dataof a local storage system to object storage in the cloud. Accordingly,certain embodiments according to the present disclosure may layer filesystem functionality on cloud object interfaces to provide cloud-basedstorage while allowing for functionality expected from a legacyapplications. For example, legacy applications, which are notcloud-based, may access data primarily as files and may be configuredfor POSIX interfaces and semantics. From the legacy applicationperspective, being able to modify content of a file without rewritingthe file is expected. Likewise, being to be able to organize data inname hierarchies is expected.

To accommodate such expectations, certain embodiments may layer POSIXinterfaces and semantics on cloud-based storage, while providing accessto data in a manner that, from a user perspective, is consistent withfile-based access with data organization in name hierarchies. Further,certain embodiments may provide for memory mapping of data so thatmemory map changes are reflected in persistent storage while ensuringconsistency between memory map changes and writes. By transforming a ZFSfile system disk-based storage into ZFS cloud-based storage, the ZFSfile system gains the elastic nature of cloud storage. By mapping “diskblocks” to cloud objects, the storage requirements for the ZFS filesystem are only the “blocks” actually in use. The system may always bethinly provisioned with no risk of running out of backing storage.Conversely, the cloud storage gains ZFS file system semantics andservices. Full POSIX semantics may be provided to cloud clients, as wellas any additional data services (such as compression, encryption,snapshots, etc.) provided by the ZFS file system.

Certain embodiments may provide the ability to migrate data to-and-fromthe cloud and may provide for local data to co-exist with data in thecloud by way of a hybrid cloud storage system, which provides forstorage elasticity and scale while layering ZFS file systemfunctionality on the cloud storage. By extending the ZFS file system toallow storing of objects in cloud object stores, a bridge may beprovided to traditional object storage while preserving ZFS file systemfunctionality in addition to all ZFS file system data services. Thebridging of the gap between traditional local file systems and theability to store data in various clouds object stores redounds tosignificant performance improvements.

Furthermore, embodiments of the present invention enable the use oftraditional ZFS data services in conjunction with the hybrid cloudstorage. As an example, compression, encryption, deduplication,snapshots, and clones are each available in certain embodiments of thepresent invention and are described in short detail immediately below.In the present invention users can continue to use all of the dataservices provided by the ZFS file system seamlessly when extendingstorage to the cloud. For instance, the Oracle Key Manager or equivalentmanages the key local to a user allowing end-to-end secure encryptionwith locally managed keys when storing to the cloud. The same commandsused to compress, encrypt, deduplicate, take snapshots, and make cloneson disk storage are used for storage to the cloud. As a result, userscontinue to benefit from the efficiencies and the security provided byZFS compression, encryption, deduplication, snapshots, and clones.

Compression is typically turned on because it reduces the resourcesrequired to store and transmit data. Computational resources areconsumed in the compression process and, usually, in the reversal of theprocess (decompression). Data compression is subject to a space—timecomplexity trade-off. For instance, a compression scheme may requireintensive processing decompression fast enough to be consumed as it isbeing decompressed. The design of data compression schemes involvestrade-offs among various factors, including the degree of compressionand the computational resources required to compress and decompress thedata.

ZFS encryption enables an end-to-end secure data block system withlocally saved ecryption keys providing an additional layer of security.ZFS encryption does not of itself prevent the data blocks from beingmissapropriated, but denies the message content to the interceptor. Inan encryption scheme, the intended data block, is encrypted using anencryption algorithm, generating ciphertext that can only be read ifdecrypted. For technical reasons, an encryption scheme usually uses apseudo-random encryption key generated by an algorithm. It is inprinciple possible to decrypt the message without possessing the key,but, for a well-designed encryption scheme, large computationalresources and skill are required. ZFS Data blocks are encrypted usingAES (Advanced Encryption Standard) with key lengths of 128, 192, and256.

Data block duplication is a specialized data compression technique foreliminating duplicate copies of repeating data blocks. Data blockdeduplication is used to improve storage utilization and can also beapplied to network data transfers to reduce the number of data blocksthat must be sent to store to memory. In the deduplication process,unique data blocks are identified and stored during a process ofanalysis. As the analysis continues, other data blocks are compared tothe stored copy and whenever a match occurs, the redundant data block isreplaced with a small reference that points to the stored stored datablock. Given that the same data block pattern may occur dozens,hundreds, or even thousands of times the number of data blocks that mustbe stored or transferred is reduced substantially using deduplication.

Snapshots for ZFS storage to the cloud object store are createdseamlessly in the ZFS system. Snapshots freeze certain data and metadatablocks so that they may not be written over in case a backup to thesnapshot is needed. A tree hierarchy can have many snapshots, and eachsnapshot will be saved until deleted. Snapshots can be stored locally orin the cloud object store. And snapshots are “free” in the ZFS system asthey don't require any extra storage capability than creating the rootblock the snapshot points to. The root block and all subsequent blocksfrom the root block are not available for copy on write operation whenaccessed from the snapshot reference to the root block. At the nextprogression after the snap shot is taken—a new root block becomes theactive root block.

Clones are created from snapshots and, unlike snapshot, blocks accessedusing the clone reference to the root block are available for copy onwrite operation. Clones allow development and trouble shooting on asystem without corrupting the active root block and tree. Clones arelinked to snapshots and snapshots cannot be deleted if a clone linkingto a snapshot block persists. Clones, in some cases, can be promoted tothe active hierarchical tree.

Various embodiments will now be discussed in greater detail withreference to the accompanying figures, beginning with FIG. 1.

FIG. 1 illustrates one example storage network 100 that may be used toimplement certain embodiments according to the present disclosure. Theselection and/or arrangement of hardware devices depicted in FIG. 1 areshown only by way of example, and are not meant to be limiting. FIG. 1provides a plurality of storage appliances 120 connected through one ormore switch circuits 122. The switch circuits 122 may connect theplurality of storage appliances 120 to a plurality of I/O servers 136,which in turn may provide access to the plurality of storage appliances120 for client devices, such as local computer systems 130, computersystems available over a network 132, and/or cloud computing systems134.

Each I/O server 136 may execute multiple independent file systeminstances, each of which may be responsible for the management of aportion of the overall storage capacity. As will be described in greaterdetail below, these file system instances may include the Oracle ZFSfile system. The I/O servers 136 may comprise blade and/or standaloneservers that include host ports 124 to communicate with the clientdevices by receiving read and/or write data access requests. The hostports 124 may communicate with an external interface provider 126 thatidentifies a correct data storage controller 128 to service each I/Orequest. The data storage controllers 128 can each exclusively manage aportion of data content in one or more of the storage appliances 120described below. Thus, each data storage controller 128 can access alogical portion of the storage pool and satisfy data requests receivedfrom the external interface providers 126 by accessing their own datacontent. Redirection through the data storage controllers 128 mayinclude redirection of each I/O request from the host ports 124 to afile system instance (e.g., a ZFS instance) executing on the I/O servers136 and responsible for the blocks requested. For example, this mayinclude a redirection from a host port 124-1 on one I/O server 136-1 toa ZFS instance on another I/O server 136-n. This redirection may allowany part of the available storage capacity to be reached from any hostport 124. The ZFS instance may then issue the necessary direct I/Otransactions to any storage device in the storage pool to complete therequest. Acknowledgements and/or data may then be forwarded back to theclient device through the originating host port 124.

A low-latency, memory-mapped network may tie together the host ports124, any file system instances, and the storage appliances 120. Thisnetwork may be implemented using one or more switch circuits 122, suchas Oracle's Sun Data Center InfiniBand Switch 36 to provide a scalable,high-performance cluster. A bus protocol, such as the PCI Express bus,may route signals within the storage network. The I/O servers 136 andthe storage appliances 120 may communicate as peers. The redirectiontraffic and ZFS memory traffic may both use the same switch fabric.

In various embodiments, many different configurations of the storageappliances 120 may be used in the network of FIG. 1. In someembodiments, the Oracle ZFS Storage Appliance series may be used. TheZFS Storage Appliance provides storage based on the Oracle Solariskernel with Oracle's ZFS file system (“ZFS”) described below. Theprocessing core 114 handles any operations required to implement anyselected data protection (e.g., mirroring, RAID-Z, etc.), data reduction(e.g., inline compression, duplication, etc.), and any other implementeddata services (e.g., remote replication, etc.). In some embodiments, theprocessing core may comprise an 8×15 core of 2.8 GHz Intel® Xeon®processors. The processing core also handles the caching of stored datain both DRAM and Flash 112. In some embodiments, the DRAM/Flash cachemay comprise a 3 TB DRAM cache.

In some configurations, the storage appliances 120 may comprise an I/Oport 116 to receive I/O requests from the data storage controllers 128.Each of the storage appliances 120 may include an integral rack-mountedunit with its own internally redundant power supply and cooling system.A concentrator board 110 or other similar hardware device may be used tointerconnect a plurality of storage devices. Active components such asmemory boards, concentrator boards 110, power supplies, and coolingdevices may be hot swappable. For example, the storage appliance 120 mayinclude flash memory 102, nonvolatile RAM (NVRAM) 104, variousconfigurations of hard disk drives 105, tape drives, RAID arrays 108 ofdisk drives, and so forth. These storage units may be designed for highavailability with hot swapping and internal redundancy of memory cards,power, cooling, and interconnect. In some embodiments the RAM may bemade non-volatile by backing it up to dedicated Flash on loss of power.The mix of Flash and NVRAM cards may be configurable, and both may usethe same connector and board profile.

Although not shown explicitly, each of the I/O servers 136 may execute aglobal management process, or data storage system manager, that maysupervise the operation of the storage system in a pseudo-static, “lowtouch” approach, intervening when capacity must be reallocated betweenZFS instances, for global Flash wear leveling, for configurationchanges, and/or for failure recovery. The “divide and conquer” strategyof dividing the capacity among individual ZFS instances may enable ahigh degree of scalability of performance, connectivity, and capacity.Additional performance may be achieved by horizontally adding more I/Oservers 136, and then assigning less capacity per ZFS instance and/orfewer ZFS instances per I/O server 136. Performance may also be scaledvertically by using faster servers. Additional host ports may be addedby filling available slots in the I/O servers 136 and then addingadditional servers. Additional capacity may also be achieved by addingadditional storage appliances 120, and allocating the new capacity tonew or existing ZFS instances.

FIG. 2 illustrates an instance of an example network file system 200that may be executed in a storage environment, including the storageenvironment of FIG. 1, in accordance with certain embodiments of thepresent disclosure. For example, the file system 200 may include theOracle ZFS file system (“ZFS”), which provides very large capacity(128-bit), data integrity, an always-consistent, on-disk format,self-optimizing performance, and real-time remote replication. Amongother ways, ZFS departs from traditional file systems at least byeliminating the need for a separate volume manager. Instead, a ZFS filesystem shares a common storage pool of storage devices and acts as boththe volume manager and the file system. Therefore, ZFS has completeknowledge of both the physical disks and volumes (including theircondition, status, and logical arrangement into volumes, along with allthe files stored on them). Devices can be added or removed from the poolas file system capacity requirements change over time to dynamicallygrow and shrink as needed without needing to repartition the underlyingstorage pool.

In certain embodiments, the system 200 may interact with an application202 through an operating system. The operating system may includefunctionality to interact with a file system, which in turn interfaceswith a storage pool. The operating system typically interfaces with thefile system 200 via a system call interface 208. The system callinterface 208 provides traditional file read, write, open, close, etc.,operations, as well as VNODE operations and VFS operations that arespecific to the VFS architecture. The system call interface 208 may actas a primary interface for interacting with the ZFS as a file system.This layer resides between a data management unit (DMU) 218 and presentsa file system abstraction of the files and directories stored therein.The system call interface 208 may be responsible for bridging the gapbetween the file system interfaces and the underlying DMU 218interfaces.

In addition to the POSIX layer of the system call interface 208, theinterface layer of the file system 200 may also provide a distributedfile system interface 210 for interacting with cluster/cloud computingdevices 204. For example, a Lustre® interface may be provided to providea file system for computer clusters ranging in size from small workgroupclusters to large-scale, multi-site clusters. A volume emulator 212 mayalso provide a mechanism for creating logical volumes which can be usedas block/character devices. The volume emulator 212 not only allows aclient system to distinguish between blocks and characters, but alsoallows the client system to specify the desired block size and therebycreate smaller, sparse volumes in a process known as “thinprovisioning.” The volume emulator 212 provides raw access 206 toexternal devices.

Underneath the interface layer lies a transactional object layer. Thislayer provides an intent log 214 configured to record a per-datasettransactional history which can be replayed upon a system crash. In ZFS,the intent log 214 saves transaction records of system calls that changethe file system in memory with sufficient information to be able toreplay the system calls. These are stored in memory until the DMU 218commits them to the storage pool and they can be discarded or they areflushed. In the event of a power failure and/or disk failure, the intentlog 214 transactions can be replayed to keep the storage pool up-to-dateand consistent.

The transactional object layer also provides an attribute processor 216that may be used to implement directories within the POSIX layer of thesystem call interface 208 by making arbitrary {key, value} associationswithin an object. The attribute processor 216 may include a module thatsits on top of the DMU 218 and may operate on objects referred to in theZFS as “ZAP objects.” ZAP objects may be used to store properties for adataset, navigate file system objects, and/or store storage poolproperties. ZAP objects may come in two forms: “microzap” objects and“fatzap” objects. Microzap objects may be a lightweight version of thefatzap objects and may provide a simple and fast lookup mechanism for asmall number of attribute entries. Fatzap objects may be better suitedfor ZAP objects containing large numbers of attributes, such as largerdirectories, longer keys, longer values, etc.

The transactional object layer also provides a data set and snapshotlayer 220 that aggregates DMU objects in a hierarchical namespace, andprovides a mechanism for describing and managing relationships betweenproperties of object sets. This allows for the inheritance ofproperties, as well as quota and reservation enforcement in the storagepool. DMU objects may include ZFS file system objects, clone objects,CFS volume objects, and snapshot objects. The data and snapshot layer220 can therefore manage snapshot and clones.

A snapshot is a read-only copy of a file system or volume. A snapshot isa view of a filesystem as it was at a particular point in time. ZFS'ssnapshots are useful in the same way that some other filesystems'ssnapshots are: By doing a backup of a snapshot, you have a consistent,non-changing target for the backup program to work with. Snapshots canalso be used to recover from recent mistakes, by copying the corruptedfiles from the snapshot. Snapshots can be created almost instantly, andthey initially consume no additional disk space within the pool.However, as data within the active dataset changes, the snapshotconsumes disk space by continuing to reference the old data, thuspreventing the disk space from being freed. The blocks containing theold data will only be freed if the snapshot is deleted. Taking asnapshot is a constant-time operation. The presence of snapshots doesn'tslow down any operations. Deleting snapshots takes time proportional tothe number of blocks that the delete will free, and is very efficient.ZFS snapshots include the following features: they persist across systemreboots; the theoretical maximum number of snapshots is 2⁶⁴; they use noseparate backing store; they consume disk space directly from the samestorage pool as the file system or volume from which they were created;recursive snapshots are created quickly as one atomic operation; andthey are created together (all at once) or not created at all. Thebenefit of atomic snapshot operations is that the snapshot data isalways taken at one consistent time, even across descendent filesystems. Snapshots cannot be accessed directly, but they can be cloned,backed up, rolled back to, and so on. Snapshots can be used to “rollback” in time to the point when the snapshot was taken

A clone is a writable volume or file system whose initial contents arethe same as the dataset from which it was created. In the ZFS systemclones are always created from snapshots. As with snapshots, creating aclone is nearly instantaneous and initially consumes no additional diskspace. In addition, you can snapshot a clone. Clones can only be createdfrom a snapshot. When a snapshot is cloned, an implicit dependency iscreated between the clone and snapshot. Even though the clone is createdsomewhere else in the dataset hierarchy, the original snapshot cannot bedestroyed as long as the clone exists. Clones do not inherit theproperties of the dataset from which it was created. A clone initiallyshares all its disk space with the original snapshot. As changes aremade to the clone, it uses more disk space. Clones are useful to branchoff and do development or troubleshooting—and can be promoted to replacethe live file system. Clones can also be used to duplicate a file systemon multiple machines.

The DMU 218 presents a transactional object model built on top of a flataddress space presented by the storage pool. The modules described aboveinteract with the DMU 218 via object sets, objects, and transactions,where objects are pieces of storage from the storage pool, such as acollection of data blocks. Each transaction through the DMU 218comprises a series of operations that are committed to the storage poolas a group. This is the mechanism whereby on-disk consistency ismaintained within the file system. Stated another way, the DMU 218 takesinstructions from the interface layer and translates those intotransaction batches. Rather than requesting data blocks and sendingsingle read/write requests, the DMU 218 can combine these into batchesof object-based transactions that can be optimized before any diskactivity occurs. Once this is done, the batches of transactions arehanded off to the storage pool layer to schedule and aggregate the rawI/O transactions required to retrieve/write the requested data blocks.As will be described below, these transactions are written on acopy-on-write (COW) basis, which eliminates the need for transactionjournaling.

The storage pool layer, or simply the “storage pool,” may be referred toas a storage pool allocator (SPA). The SPA provides public interfaces tomanipulate storage pool configuration. These interfaces can create,destroy, import, export, and pool various storage media and manage thenamespace of the storage pool. In some embodiments, the SPA may includean adaptive replacement cache (ARC) 222 that acts as a central point formemory management for the SPA. Traditionally, an ARC provides a basicleast-recently-used (LRU) object replacement algorithm for cachemanagement. In ZFS, the ARC 222 comprises a self-tuning cache that canadjust based on the I/O workload. Additionally, the ARC 222 defines adata virtual address (DVA) that is used by the DMU 218. In someembodiments, the ARC 222 has the ability to evict memory buffers fromthe cache as a result of memory pressure to maintain a high throughput.

The SPA may also include an I/O pipeline 224, or “I/O manager,” thattranslates the DVAs from the ARC 222 into logical locations in each ofthe virtual devices (VDEVs) 226 described below. The I/O pipeline 224drives the dynamic striping, compression, checksum capabilities, anddata redundancy across the active VDEVs. Although not shown explicitlyin FIG. 2, the I/O pipeline 224 may include other modules that may beused by the SPA to read data from and/or write data to the storage pool.For example, the I/O pipeline 224 may include, without limitation, acompression module, an encryption module, a checksum module, and ametaslab allocator. The checksum may be used, for example, to ensuredata has not been corrupted. In some embodiments, the SPA may use themetaslab allocator to manage the allocation of storage space in thestorage pool.

Compression is the process of reducing the data size of a data block(referred to interchangeably with leaf node or data node), typically byexploiting redundancies in the data block itself. Many differentcompression types are used by ZFS. When compression is enabled, lessstorage can be allocated for each data block. The following compressionalgorithms are available. LZ4—an algorithm added after feature flagswere created. It is significantly superior to LZJB. LZJB is the originaldefault compression algorithm) for ZFS. It was created to satisfy thedesire for a compression algorithm suitable for use in filesystems.Specifically, that it provides fair compression, has a high compressionspeed, has a high decompression speed and detects incompressible datadetection quickly. GZIP (1 through 9 implemented in the classicLempel-Ziv implementation. It provides high compression, but it oftenmakes IO CPU-bound. ZLE (Zero Length Encoding)—a very simple algorithmthat only compresses zeroes. In each of these cases there is a trade-offof compression ratio to the amount of latency involved in compressingand decompressing the data block. Typically—the more compressed thedata—the longer it takes to compress and decompress it.

Encryption is the process of adding end-to-end security to data blocksby encoding them cryptographically with a key. Only users with a key candecrypt the data block. As used in the ZFS system, a ZFS pool cansupport a mix of encrypted and unencrypted ZFS data sets (file systemsand ZVOLs). Data encryption is completely transparent to applicationsand provides a very flexible system for securing data at rest, and itdoesn't require any application changes or qualification. FurthermoreZFS encryption randomly generates a local encryption key from apassphrase or an AES key and all keys are stored locally with theclient—not in the cloud object store 404 as traditional file systems do.Encryption is transparent to the application and storage to the cloudobject store 404 when turned on. ZFS makes it easy to encrypt data andmanage data encryption. You can have both encrypted and unencrypted filesystems in the same storage pool. You can also use different encryptionkeys for different systems, and you can manage encryption either locallyor remotely—although the randomly generated encryption key alwaysremains local. ZFS encryption is inheritable to descendent file systems.Data is encrypted using AES (Advanced Encryption Standard) with keylengths of 128, 192, and 256 in the CCM and GCM operation modes.

Deduplication is the process of recognizing that a data block to bestored in the file system is already stored on the file system as anexisting data block and pointing to that existing data block rather thanstoring the data block again. ZFS provides block-level deduplicationbecause this is the finest granularity that makes sense for ageneral-purpose storage system. Block-level dedup also maps naturally toZFS's 256-bit block checksums, which provide unique block signatures forall blocks in a storage pool as long as the checksum function iscryptographically strong (e.g. SHA256). Deduplication is synchronous andis performed as data blocks are sent to the cloud object store 404. Ifdata blocks are not duplicated, enabling deduplication will add overheadwithout providing any benefit. If there are duplicate data blocks,enabling deduplication will both save space and increase performance.The space savings are obvious; the performance improvement is due to theelimination of storage writes when storing duplicate data, plus thereduced memory footprint due to many applications sharing the same pagesof memory. Most storage environments contain a mix of data that ismostly unique and data that is mostly replicated. ZFS deduplication isper-dataset and can be enabled when it is likely to help.

In ZFS, the storage pools may be made up of a collection of VDEVs. Incertain embodiments, at least a portion of the storage pools may berepresented as a self-described Merkle tree, a logical tree where bothdata and metadata are stored by VDEVs of the logical tree. There are twotypes of virtual devices: physical virtual devices called leaf VDEVs,and logical virtual devices called interior VDEVs. A physical VDEV mayinclude a writeable media block device, such as a hard disk or Flashdrive. A logical VDEV is a conceptual grouping of physical VDEVs. VDEVscan be arranged in a tree with physical VDEVs existing as leaves of thetree. The storage pool may have a special logical VDEV called a “rootVDEV” which roots the tree. All direct children of the root VDEV(physical or logical) are called “top-level” VDEVs. In general, VDEVsimplement data replication, mirroring, and architectures such as RAID-Zand RAID-Z2. Each leaf VDEV represents one or more physical storagedevices 228 that actually store the data provided by the file system.

In some embodiments, the file system 200 may include an object-basedfile system where both data and metadata are stored as objects. Morespecifically, the file system 200 may include functionality to storeboth data and corresponding metadata in the storage pool. A request toperform a particular operation (i.e., a transaction) is forwarded fromthe operating system, via the system call interface 208, to the DMU 218,which translates the request to perform an operation on an objectdirectly to a request to perform a read or write operation (i.e., an I/Orequest) at a physical location within the storage pool. The SPAreceives the request from the DMU 218 and writes the blocks into thestorage pool using a COW procedure. COW transactions may be performedfor a data write request to a file. Instead of overwriting existingblocks on a write operation, write requests cause new segments to beallocated for the modified data. Thus, retrieved data blocks andcorresponding metadata are never overwritten until a modified version ofthe data block and metadata are committed. Thus, the DMU 218 writes allthe modified data blocks to unused segments within the storage pool andsubsequently writes corresponding block pointers to unused segmentswithin the storage pool. To complete a COW transaction, the SPA issuesan I/O request to reference the modified data block.

FIGS. 3A-3D illustrate a COW process for a file system, such as the filesystem 200, in accordance with certain embodiments of the presentdisclosure. For example, the ZFS system described above uses a COWtransactional model where all block pointers within the file system maycontain 256-bit checksum of a target block which is verified when theblock is read. As described above, blocks containing active data are notoverwritten in place. Instead the new block is allocated, modified datais written to it, and then any metadata blocks referencing it are simplyread, reallocated, and rewritten.

FIG. 3A illustrates a simplified diagram of a file system storage ofdata and metadata corresponding to one or more files as a logical tree300, according to some embodiments. The logical tree 300, as well asother logical trees described herein, may be a self-described Merkletree where the data and metadata are stored as blocks of the logicaltree 300. A root block 302 may represent the root of the logical tree300, or “uberblock.” The logical tree 300 can be traversed through filesand directories by navigating through each child node 304, 306 of theroot 302. Each non-leaf node represents a directory or file, such asnodes 308, 310, 312, and 314. In some embodiments, each non-leaf nodemay be assigned a hash of values of its child nodes. Each leaf node 316,318, 320, 322 represents a data block of a file.

FIG. 3B illustrates an example of the logical tree 300-1 after aninitial stage of a write operation. In this example, the data blocksrepresented by nodes 324 and 326 have been written by the file system200. Instead of overwriting the data in nodes 316 and 318, new datablocks are allocated for nodes 324 and 326. Thus, after this operation,the old data in nodes 316 and 318 persist in the memory along with thenew data in nodes 324 and 326.

FIG. 3C illustrates an example of the logical tree 300-2 as the writeoperation continues. In order to reference the newly written data blocksin nodes 324 and 326, the file system 200 determines nodes 308 and 310that reference the old nodes 316 and 318. New nodes 328 and 330 areallocated to reference the new data blocks in nodes 324 326. The sameprocess is repeated recursively upwards through the file systemhierarchy until each node referencing a changed node is reallocated topoint to the new nodes.

When the pointer blocks are allocated in new nodes in the hierarchy, theaddress pointer in each node is updated to point to the new location ofthe allocated child in memory. Additionally, each data block includes achecksum that is calculated by the data block referenced by the addresspointer. For example, the checksum in node 328 is calculated using thedata block in node 324. This arrangement means that the checksum isstored separately from the data block from which it is calculated. Thisprevents so-called “ghost writes” where new data is never written, but achecksum stored with the data block would indicate that the block wascorrect. The integrity of the logical tree 300 can be quickly checked bytraversing the logical tree 300 and calculating checksums at each levelbased on child nodes.

In order to finalize the write operation, the root 302 can bereallocated and updated. FIG. 3D illustrates an example of the logicaltree 300-3 at the conclusion of the write operation. When the root 302is ready to be updated, a new uberblock root 336 can be allocated andinitialized to point to the newly allocated child nodes 332 and 334. Theroot 336 can then be made the root of the logical tree 300-3 in anatomic operation to finalize the state of the logical tree 300-3.

A snapshot is a read-only copy of a file system or volume. A snapshot isa view of a filesystem as it was at a particular point in time. ZFS'ssnapshots are useful in the same way that some other file systems'snapshots are: by doing a backup of a snapshot, you have a consistent,non-changing target for the backup program to work with. Snapshots canalso be used to recover from recent mistakes, by copying the corruptedfiles from the snapshot. Snapshots can be created almost instantly, andthey initially consume no additional disk space within the pool.However, as data within the active dataset changes, the snapshotconsumes disk space by continuing to reference the old data, thuspreventing the disk space from being freed. The blocks containing theold data will only be freed if the snapshot is deleted. Taking asnapshot is a constant-time operation. The presence of snapshots doesnot slow down any operations. Deleting snapshots takes time proportionalto the number of blocks that the delete will free, and is veryefficient. ZFS snapshots include the following features: they persistacross system reboots; the theoretical maximum number of snapshots is2⁶⁴; they use no separate backing store; they consume disk spacedirectly from the same storage pool as the file system or volume fromwhich they were created; recursive snapshots are created quickly as oneatomic operation; and they are created together (all at once) or notcreated at all. The benefit of atomic snapshot operations is that thesnapshot data is always taken at one consistent time, even acrossdescendent file systems. Snapshots cannot be accessed directly, but theycan be cloned, backed up, rolled back to, and so on. Snapshots can beused to “roll back” in time to the point when the snapshot was taken.FIG. 3E depicts an example of the snapshot data service in ZFS where thesnapshot was taken before the COW process described in FIGS. 3A-3D maderoot block 336 the new live root block. A live root block is the rootblock that the next data progression will be made from when performing aCOW. The snapshot root and the “live” root are shown. The live root isthe root that will be operated on in the next storage operation. Allblocks pointed to by the snapshot root (302-322) are made “read only”meaning that they are put on list of blocks that cannot be freed forfurther use by the storage system until the snapshot is deleted.

It should be noted that some terminology is used interchangeablythroughout the application. For instance leaf nodes, leaf blocks, anddata blocks may be the same in certain instances, particularly whenreferencing local tree instances. Further, non-leaf nodes, metadata, andmetadata blocks may be used interchangeably in certain instances,particularly when referencing local tree instances. Root nodes and rootblocks similarly may be used interchangeably in certain instances,particularly when referencing local tree instances. Further, it shouldbe noted that references to leaf nodes, non-leaf nodes, and root nodesmay be similarly applied to cloud storage objects corresponding to cloudversions of logical trees generated based at least in part on local treeinstances.

When blocks are created they are given a “birth” time that representsthe iteration or progression of the COW that the block was created. FIG.3F demonstrates this idea. In FIG. 3F, as shown in 365—the birth time 19of FIG. 3A. 366 shows the birth time, 25 of FIG. 3D. 367 shows a birthtime 37 as shown by the new tree created by blocks 372, 378, 379, 384,386, and 392, and represents a data transaction on the tree 12iterations after birth time 25. Thus a rollback or back up to thesnapshot would leave only the blocks as shown in FIG. 3A. Thus—using abirth time hierarchy—the ZFS system can generate and roll back to anypoint in a birth time for the entire tree structure from a snapshot ofthe root of the tree. Essentially, this allows all new blocks with birthtimes after the snapshot to be made available in the storage pool aslong as they are not linked to by any other snapshot or metadata block.

A clone is a writable volume or file system whose initial contents arethe same as the dataset from which it was created. In the ZFS system,clones are always created from snapshots. As with snapshots, creating aclone is nearly instantaneous and initially consumes no additional diskspace. In addition, you can snapshot a clone. Clones can only be createdfrom a snapshot. When a snapshot is cloned, an implicit dependency iscreated between the clone and snapshot. Even though the clone is createdsomewhere else in the dataset hierarchy, the original snapshot cannot bedestroyed as long as the clone exists. Clones do not inherit theproperties of the dataset from which it was created. A clone initiallyshares all its disk space with the original snapshot. As changes aremade to the clone, it uses more disk space. Clones are useful to branchoff and do development or troubleshooting—and can be promoted to replacethe live file system. Clones can also be used to duplicate a file systemon multiple machines.

The embodiments described herein may be implemented in the systemdescribed above in FIGS. 1-3. For example, the system may comprise oneor more processors of the various servers, storage appliances, and/orswitching circuits of FIG. 1. Instructions may be stored in one or morememory devices of the system that cause the one or more processors toperform various operations that affect the functioning of the filesystem. Steps of various methods may be performed by the processors,memory devices, interfaces, and/or circuitry of the system in FIGS. 1-2.

Turning now to FIG. 4, FIG. 4 is a high-level diagram illustrating anexample of a hybrid cloud storage system 400, in accordance with certainembodiments of the present disclosure. The hybrid cloud storage system400 may transform a network file system, such as a ZFS file system, intoa cloud-capable file system where functionality of the file system,including file system data services, is layered on a cloud object storethat is remote from the file system. As in the depicted diagram, thehybrid cloud storage system 400 may include the network file system 200(also referenced herein as the “local file system 200”). The local filesystem 200 may be communicatively coupled to a cloud object storage 404.In some embodiments, the cloud object storage 404 may correspond to thecluster/cloud 204 indicated in FIG. 2. The local file system 200 may becommunicatively coupled to the cloud object storage 404 by way of acloud interface appliance 402. The cloud interface appliance 402 may beused by the local file system 200 as an access point for the cloudobject store 404.

The hybrid cloud storage system 400 provides a solution to overcometraditional limitations of cloud object storage. Traditional cloudobject protocols are limited to restricted data/object access semantics.Cloud object stores traditionally have limited interfaces andprimitives, and are not POSIX compliant. For example, once an object iswritten, it cannot be thereafter modified; it may only be deleted andreplaced with a newly created object. As another example, traditionalcloud object storage has namespace limitations such that the namespaceis simplified and limited to only top-level containers. However, notonly may the hybrid cloud storage system 400 migrate data to and fromthe cloud object store 404, but also the hybrid cloud storage system 400may layer file system functionality of the local file system 200 oncloud object interfaces to the cloud object storage 404 to provide thecloud-based storage.

The local file system 200 may be configured for POSIX interfaces andsemantics. For example, the local file system 200 may provide a userwith access to data as files, allowing for modification of content of afile without rewriting the file. The local file system 200 may alsoprovide for the organization of data in name hierarchies as is typicalfor ZFS file systems. All the functionalities of a ZFS file system maybe available to a user of the local file system 200. The cloud interfaceappliance 402 may allow for layering of file-system semantics on top ofa cloud object protocol—for example, to provide the abilities toconstruct a namespace, create files, create directories, etc.—and extendsuch abilities with respect to data migrated to and from the cloudobject storage 404. The cloud interface appliance 402 may facilitate aplug-n-play object storage solution to improve the local file system 200while supporting ZFS file system data services.

The cloud interface appliance 402 may be configured to provide an objectAPI (application programming interface). In some embodiments, the cloudinterface appliance 402 may be configured to use a number of APItranslation profiles. According to certain embodiments, the APItranslation profiles may integrate modules and functions (e.g., the dataservices and modules), POSIX interfaces and semantics, and othercomponents which may not be natively designed to interact with cloudstorage. The API translation profiles, in some embodiments, maytranslate protocols, formats, and routines of the file system 200 (e.g.,by way of API calls) to allow interaction with the cloud data store 404.Information for such integration may be stored in an API translationdata store, which could be co-located with the cloud interface appliance402 or otherwise communicatively coupled to the cloud interfaceappliance 402. The cloud interface appliance 402 may utilize theinformation to cohesively integrate POSIX interfaces and semantics tointerface with the cloud data store 404 while preserving the semantics.

The hybrid cloud storage system 400 may allow the local file system 200to use the cloud object storage 404 as a “drive.” In various instances,the files 410 may be stored as data objects with metadata objects,and/or as data blocks with associated metadata. The cloud interfaceappliance 402 may receive and transfer files 410 from and to the localfile system 200. In various embodiments, the local file system 200 mayreceive and/or transmit the files 410 via the NFS (Network File System)protocol, SMB (Server Message Block Protocol), and/or the like. In someembodiments, the cloud interface appliance 402 may translate the files410 into objects 412. The translation of the files 410 may includetranslating data blocks and associated metadata and/or data objectsassociated with metadata objects, any of which may correspond to thefiles 410. In some embodiments, the translation may include the cloudinterface appliance 402 performing API translation with a number of theAPI translation profiles. The translation, according to someembodiments, may include the cloud interface appliance 402 extractingdata and/or metadata from the files 410, objects, and/or blocks. Thecloud interface appliance 402 may convert the files 410, objects, and/orblocks to cloud storage objects at least in part by using the extracteddata. In some embodiments, the cloud interface appliance 402 may createcorresponding cloud storage objects with extracted data embedded in putrequests directed to the cloud object store 404. Likewise, with any ofthe translations effected by the cloud interface appliance 402 tointerface out to the cloud data store 404, the cloud interface appliance402 may, in some embodiments, reverse the translation processes tointerface with local components of the local file system 200.

The cloud interface appliance 402 may transfer and receive objects 412to and from the cloud object storage 404. In some embodiments, the cloudinterface appliance 402 may transceive the objects 412 via HTTPS and/orthe like. In some embodiments, the cloud interface appliance 402 may beco-located with the local file system 200. In other embodiments, thecloud interface appliance 402 may be located remotely from local filesystem 200, such as with at least some equipment facilitating the cloudobject store 404 or at some other communicatively coupled site.

As disclosed further herein, files in the local file system 200 may bestored as “disk blocks,” virtual storage blocks where data objects andmetadata corresponding to a file are stored as a logical tree 300 (e.g.,a self-described Merkle tree where data and metadata are stored asblocks). The cloud interface appliance 402 may create a mapping 406 ofeach logical block in the tree 300 of data directly to cloud objects 414in the cloud object store 404. Some embodiments may employ a one-to-oneblock-to-object mapping. Other embodiments, additionally oralternatively, may employ any other suitable ratio of blocks to cloudobjects, for example, to map multiple blocks to one cloud object. Insome instances of such embodiments, an entire logical tree of blocks maybe mapped to a single cloud object. In other instances, only part of alogical tree of blocks may be mapped to a single cloud object.

In some embodiments, address pointers are updated when the blocks areconverted into cloud objects. When blocks are converted into cloudobjects with a one-to-one block-to-object conversion scheme, the addresspointers may be updated so that non-leaf cloud objects in the hierarchypoint to child cloud objects in the cloud object store 404. By way ofexample, an address pointer could correspond to an object name of childcloud object and path specification, which may include parameters suchas the object name, a bucket specification, etc. Accordingly, someembodiments may translate the blocks of the logical tree 300 to cloudobjects of the logical tree 300A. With some embodiments, suchtranslation may enable the cloud interface appliance 402 to traverse thelogical tree 300A of cloud objects utilizing the address pointers of thecloud objects.

In some embodiments, when blocks are converted into cloud objects with amultiple-to-one block-to-object conversion scheme such that part of thelogical tree 300 is converted to one cloud object, the address pointersthe cloud objects comprising the logical 300A may be similarly updated,but to a less granular extent, so that non-leaf cloud objects in thehierarchy point to child cloud objects in the cloud object store 404.Such translation may enable the cloud interface appliance 402 totraverse the logical tree 300A of cloud objects utilizing the addresspointers of the cloud objects, in a less granular but faster manner thantraversal facilitated by the one-to-one block-to-object conversionscheme. Additionally, in some embodiments, checksums may be updated withconversion process. Checksums for individual cloud objects could beupdated and stored separately in parent cloud objects. In conversionsemploying the multiple-to-one block-to-object conversion scheme, asingle checksum could be calculated for a cloud object that correspondsto a set of blocks.

The realization of the mapping 406 accordingly allows for communicationsover one or more networks to the cloud object store 404, and theinterface with the cloud object store 404 may be object-based as opposedto block-based. As disclosed further herein, with the cloud interfaceappliance 402 between the local file system 200 and the cloud objectstore 404, the hybrid cloud storage system 400 may possess differentcharacteristics and failure modes than traditional ZFS file systems. Thecloud interface appliance 402 may translate file system interfaces ofthe local file system 202 on the client side and may be capable ofcoordination via object protocol out to the cloud object store 404 toread and write data. Through the cloud interface appliance 402, thecloud objects 414 may remain accessible by the local file system 200over NF S, SMB, and/or the like.

With the mapping 406 of cloud objects 414 as logical blocks, collectionsof the cloud objects 414 may be grouped to form a drive that hosts ZFSstorage pools as self-contained collections. The drive content may beelastic such that cloud objects may only be created for logical blocksthat have been allocated. In some embodiments, the cloud interfaceappliance 402 may have the capability to assign variable object sizes(e.g., for different data types) to allow for greater storageflexibility. The data need not be limited to a specific byte size.Storage size may be expanded as needed by modifying the metadata. Insome embodiments, cloud-based pools may be imported on any server. Onceimported, cloud-based pools may appear as local storage with all ZFSservices supported for the cloud-based pools. A cloud-based pool may beindicated as a new type of storage pool. Yet, from a user perspective,the data from the cloud-based pools may appear indistinguishable fromnative pools.

FIG. 5 illustrates an instance of an example network file system 200-1of the hybrid cloud storage system 400, in accordance with certainembodiments of the present disclosure. The file system 200-1 maycorrespond to the file system 200, but with cloud device managementintegrated directly into a ZFS control stack. Beyond that which isdisclosed with respect to the file system 200, the file system 200-1 mayinclude a cloud interface device 502 that facilitates leveraging of thecloud object store 404 as a storage medium for the file system 200-1.The cloud interface device 502 may facilitate a cloud drive at least inpart by mapping a cloud storage into a device abstraction.

In some embodiments, the cloud interface device 502 may correspond toone or more VDEVs of another VDEV type of a device driver interfaceinside a ZFS file system architecture. The ZFS may communicate directlywith the cloud interface device 502. The cloud interface device 502 maybe at a virtual device layer directly above a driver layer of the filesystem 200-1. Some embodiments of the cloud interface device 502 maycorrespond to an abstraction of the device driver interface inside theZFS architecture. Other components of the file system 200-1 maycommunicate with the cloud interface device 502 as though it was anotherVDEV of another device type, such as the VDEVs 226. To enable passage ofa greater amount of information through the cloud interface device 502relative to through other VDEVs 226, interfaces associated with thecloud interface device 502 may be wider to pass more information throughthe I/O pipeline 224 and the cloud interface device 502, out to thecloud object data store 404.

In some embodiments, the cloud interface device 502 may translate filesystem interfaces on the client end. In some embodiments, in order toprovide complete POSIX file system semantics, the cloud interface device502 may convert file system interface requests into object interfacerequests directed toward the cloud object store 404. In someembodiments, the cloud interface device 502 may be capable ofcommunicating via object protocol out to the cloud object store 404 toread and write data.

FIG. 6 is a diagram illustrating additional aspects of a cloud interfaceappliance 402-1 of a hybrid cloud storage system 400-1, in accordancewith certain embodiments of the present disclosure. As indicated in theexample depicted, some embodiments of the cloud interface appliance 402may include a virtual storage pool 602 and a cloud interface daemon 604.The virtual storage pool 602 may be at the kernel of the file system200-1, and the cloud interface daemon 604 may be on the user space ofthe file system 200-1. In various embodiments, the cloud interfacedaemon 604 may correspond to a cloud interface component of theapplication 202 and/or the cluster/cloud 204 indicated in FIG. 5.

In some embodiments, the virtual storage pool 602 may include at leastone cloud interface device 502, intent log 214-2, and cache 222-1. Theintent log 214-2 and the cache 222-1 described above with respect toFIGS. 1-2 and below with respect to FIG. 7. The cloud interface device502 may interact with the cloud interface daemon 604 to coordinateoperations with respect to the cloud object data store 404 based atleast in part on the mapping 406. The cloud interface daemon 604 mayinclude a cloud client interface 608 to interface with the cloud objectdata store 404. In some implementations, the cloud client interface 608may include an endpoint providing Swift/S3 compatibility with the cloudobject data store 404. Operations of the cloud client interface 608 maybe based at least in part on getting and putting whole data objects 412in order to facilitate read and write access to the cloud object datastore 404.

Referring back to FIGS. 4 and 5, in operation according to someembodiments, requests to perform one or more transactions with respectto one or more files may be received from the application 202 at anapplication layer of the file system 200-1, and through the system callinterface 208 of the interface layer of the file system 200-1. Therequests may be POSIX-compliant and may be converted by one or morecomponents of the file system 200-1 into one or more object interfacerequests to perform one or more operations with respect to a cloud-basedinstantiation 300A of the logical tree 300 stored in the cloud objectstore 404. For example, in some embodiments, the cloud interfaceappliance 402 may convert the POSIX-compliant requests, or intermediaryrequested caused by the POSIX-compliant requests, into correspondingobject interface requests. In some embodiments, the DMU 218 maytranslate the POSIX-compliant requests into I/O requests to perform I/Ooperations, and the cloud interface appliance 402 may translate the I/Orequests into corresponding object interface requests, coordinating theobject interface requests using the mapping 406.

In some instances, the transactions could correspond to operations tocause storage of the files locally. In some instances, the file system200-1 may store data objects and corresponding metadata in a systemstorage pool 416 provided by the one or more of the VDEVs 226 and theone or more physical storage devices 228. The data objects maycorrespond to the one or more files. As disclosed above, the dataobjects and metadata corresponding to the one or more files may bestored as a logical tree 300. Hence, the storage of the logical tree 300may be stored locally in the system storage pool 416.

In further operation according to some embodiments, the file system200-1 may cause storage of the data objects and the correspondingmetadata of the logical tree 300 in the cloud object store 404. Whilethe logical tree 300 may first be stored in the local storage poolbefore being migrated to cloud storage in some embodiments, in otherembodiments the logical tree 300 may not be stored in the local storagepool before being stored in the cloud object store 404. For example,some embodiments may create at least part of the logical tree 300 incache and then migrate it to the cloud object store 404. Thus, it shouldbe appreciated that various embodiments are possible.

In order to store the data objects and the corresponding metadata of thelogical tree 300 in the cloud object store 404, the cloud interfacedevice 502 may create a mapping 406 of each logical block in the logicaltree 300 to a respective cloud object 414 in the cloud object store 404.In some embodiments, the DMU 218 may read data from the system storagepool 416 (e.g., from a RAIDZ or a RAIDZ2 of the local pool) to providethe data to the cloud interface device 502 as a basis for creating themapping 406. In some embodiments, the cloud interface device 502 maycommunicate directly or indirectly with another VDEV 226 to read data asa basis for the mapping 406. In some embodiments, the mapping 406 maymap objects directly to blocks represented in a physical drive. Themapping 406 may be more refined than mapping file parts to objects; itmay map at a lower level. Accordingly, the mapping 406 may be aper-object mapping 406. The mapping 406 may map virtual storage blocksonto objects in the cloud object store 404 so that the logical tree 300is represented in the cloud object store 404, as is illustrated by thelogical tree 300A. When the local file system 200-1 interfaces with thedata objects 416 in the cloud object store 404, the logical tree 300Aconforms to a new device type with which the local file system 200-1 isable to communicate.

The mapping 406 may be updated with every I/O operation or only withwrite operations to the cloud object store 404. In some embodiments, themapping 406 may include an object directory that indexes all the cloudobjects 414. Cloud object states may be kept in an index, a table, anindex-organized table, and/or the like which may be indexed on aper-object basis. In some embodiments, the mapping 406 may include anobject directory that indexes only some of the cloud objects 414. Forexample, such embodiments may index only cloud objects 414 thatcorrespond to uberblocks. In some embodiments, cloud object states foreach object relative to each leaf path may be indexed. In someembodiments, the object directory may reference the cloud objects 414 byaddress pointers that could correspond to object names and pathspecifications. In some embodiments, the object directory may referencethe cloud objects 414 by URLs.

The cloud object states indexed in the mapping 406 may be used to routeobject requests. Utilizing the index, the cloud interface device 502 mayrequest cloud objects based at least in part on the uberblocks.According to a first method, such requests may entail requesting a setof cloud objects associated with a particular uberblock so that theentire logical tree 300A is represented by the set of cloud objectstransferred in response to the request. According to a second method,such requests may entail iterative requests for subsets of cloud objectsassociated with a particular uberblock in order to iteratively traversethe entire logical tree 300A until the desired one or more cloud objectsare read from the cloud object store 404. With certain embodiments, thecloud interface device 502 may selectively used one of the two methodsbased at least in part on the size of the cloud objects representingvarious logical trees 300A. For example, the cloud interface device 502could utilize one method when the size of the cloud objects is less thanan aggregate size threshold, and transition to the other method when thesize of the cloud objects meets or exceeds the aggregate size threshold.

Some embodiments may employ another method where the object directorymay index the cloud objects on a per-object basis and may be used torequest cloud objects directly without tree traversal at the cloudlevel. Some embodiments may retain a local snapshot of metadata of thelogical tree 300A. Such embodiments may utilize the local snapshot torequest cloud objects directly or indirectly. Additionally, someembodiments may retain checksums for the logical tree 300A in the objectdirectory or a local snapshots, which checksums may be used to validatecloud objects retrieved from the cloud data store 404.

Thus, the file system 200-1 may maintain a tree of data and map thattree onto the cloud object store 404. The namespace of the tree 300A maycorrespond to metadata stored within nodes of the tree 300A. The filesystem 200-1 may continue to use a hierarchical tree representation, butmap the hierarchical tree representation into a cloud object store as away to store the data.

Referring again to FIG. 6, to effect I/O operations with respect to thecloud object store 404, the cloud interface device 502 may send requeststo the cloud interface daemon 604. For example, in some implementations,the cloud interface device 502 may send requests through thetransactional object layer and the interface layer of the file system200-1 to the cloud interface daemon 604. The requests sent by the cloudinterface device 502 may be based at least in part on POSIX-compliantrequests received via the application 202 and/or based at least in parton I/O requests created by the DMU 218 (e.g., responsive toPOSIX-compliant requests), which the cloud interface device 502 mayconvert into the requests for the cloud interface daemon 604.

In some embodiments, the requests sent by the cloud interface device 502to the cloud interface daemon 604 may be translated into get requestsand put requests for the cloud client interface 608. The requests sentby the cloud interface device 502 may be get requests and put requestsin some embodiments; in other embodiments, the cloud interface daemon604 may translate the requests sent by the cloud interface device 502into get requests and put requests. In any case, responsive to therequests sent by the cloud interface device 502, the cloud interfacedaemon 604 may communicate, via the object protocol over the one or morenetworks, with the cloud object store 404 to perform corresponding I/Ooperations with respect to the data objects 414.

For example, the communications to the cloud object store 404 mayinclude specifying, based at least in part on the mapping 406, storageof the data objects and the corresponding metadata of the logical tree300A in the cloud object store 404. In some embodiments, thecommunications may specify different object sizes, for example, fordifferent data types. Thus, the cloud interface appliance 402 mayspecify a certain object size to store certain data objects that areidentified as being of one data type, and may specify a different objectsize to store other data objects that are identified as being of adifferent data type.

FIG. 7A is a block diagram that illustrates an example method 700directed to certain features of a COW process for the hybrid cloudstorage system 400, in accordance with certain embodiments of thepresent disclosure. According to certain embodiments, the method 700 maybegin as indicated by block 702. However, teachings of the presentdisclosure may be implemented in a variety of configurations. As such,the order of certain steps comprising the method 700 and/or othermethods disclosed herein may be shuffled or combined in any suitablemanner and may depend on the implementation chosen. Moreover, while thefollowing steps may be separated for the sake of description, it shouldbe understood that certain steps may be performed simultaneously orsubstantially simultaneously.

As indicated by block 702, POSIX-compliant request(s) to performparticular operation(s) (i.e., a transaction(s)) may be received fromthe application 202. Such an operation may correspond to writing and/ormodifying data. As indicated by block 704, the POSIX-compliantrequest(s) may be forwarded from the operating system, via the systemcall interface 208, to the DMU 218. In various embodiments, transactionseffected through the DMU 218 may include a series of operations that arecommitted to one or both of the system storage pool 416 and the cloudobject store 404 as a group. These transactions may be written on a COWbasis.

As indicated by block 706, the DMU 218 may translate requests to performoperations on data objects directly to requests to perform writeoperations (i.e., I/O requests) directed to a physical location withinthe system storage pool 416 and/or the cloud object store 404. In somemodes, the operations may be performed on locally stored data objectsfirst, then changes to the data objects may be propagated tocorresponding cloud-stored data objects, by the DMU 218 directing thespecific changes or by the DMU 218 directed the cloud interfaceappliance 402 to read the changes directly or indirectly with anotherVDEV 226. In other modes, the operations may be performed on locallystored data objects and corresponding cloud-stored data objectssimultaneously or substantially simultaneously. In still other modes,the operations may be performed on cloud-stored data objects only. Forexample, some implementations may not have local tree 300, and may onlyhave a cloud-based version of a logical tree 300A. Various embodimentsmay be configured to allow user selection of one or more modes.

As indicated by block 708, the SPA may receive the I/O requests from theDMU 218. And, responsive to the requests, the SPA may initiate writingof data objects into the system storage pool 416 and/or the cloud objectstore 404 using a COW procedure. As indicated by block 710, in modeswhere writing of data objects is performed on locally stored dataobjects before, or concurrently with, writing of data objects to thecloud object store 404, the COW procedure disclosed above (e.g., in viewof FIGS. 3A-3D) may proceed with respect to the system storage pool 416.

As indicated by block 712, the cloud interface appliance 402 may receivethe I/O requests and identify incremental modifications to the logicaltree 300A. The incremental modifications may correspond to new treeportions resulting from COW processes to effect the write requests. Thecloud interface appliance 402 may translate the I/O requests intocorresponding object interface requests. Having the modified data eitherfrom I/O requests or from reading changes to locally stored dataobjects, the cloud interface device 502 may coordinate the objectinterface requests using the mapping 406 of cloud storage objects 414 inthe cloud object store 404.

For example, in some embodiments, the incremental modifications may bedetermined based at least in part on changes to the data objects storedlocally to reflect changes to the logical tree 300. In some instances,the cloud interface appliance 402 may read the logical tree 300 or atleast the changes thereto in order to determine the incrementalmodifications. Such data may be passed to the cloud interface appliance402 by another component of the file system such as the DMU 218 or amirror VDEV. With some embodiments, the incremental modifications may betransferred to the cloud interface appliance 402. However, in someembodiments, the cloud interface appliance 402 may determine theincremental modifications based at least in part on analyzing writerequests in view of a copy or a snapshot of the logical tree 300 and/ora snapshot of the logical tree 300A. With embodiments where the cloudinterface appliance 402 uses a snapshot of the logical tree 300A, thesnapshot may, in some embodiments, be retained in the mapping 406 orotherwise be stored in memory and/or the physical layer.

Referring again more particularly to FIG. 7A, as indicated by block 714,the determination of the incremental modifications may include creatingnew leaf nodes (e.g., leaf nodes 324, 326). After an initial stage of awrite operation, new data blocks (i.e., leaf nodes) have been allocatedin memory and data per the write operation has been written by the cloudinterface appliance 402 to the new data blocks, while the previous dataand data blocks likewise persist in memory.

As indicated by block 729, after the leaf nodes (data blocks) arecreated in block 714, a determination is made as to whether any of theZFS data services have been turned on or requested. These include, butare not limited to, compression, encryption, deduplication,snapshotting, and cloning. If any of these data services are notrequired—then as indicated by block 716, new non-leaf nodes may becreated (e.g., non-leaf nodes 326, 330). As the write operationcontinues, the cloud interface appliance 402 may determine non-leafnodes that reference previous versions of nodes. In order to referencethe newly written data, new non-leaf nodes are allocated to referencethe new data blocks in the leaf nodes. The same process may be repeatedrecursively upwards through the hierarchy of the logical tree 300A,reflected by the snapshot 301, until each non-leaf node referencing achanged node is reallocated to point to new nodes. When the pointerblocks are allocated in new nodes in the hierarchy, the address pointerin each node may be updated to point to the new location of theallocated child in memory. As indicated by block 718, in order tofinalize the write operation, the root node can be reallocated andupdated (e.g., root node 336). When the root node is ready to beupdated, a new root node (uberblock) can be allocated and initialized topoint to the newly allocated child nodes below the new root node.

As part of the writing operation, the metadata of all portions of thetree involved in transactional write operations is updated with checksumming. Each node created includes a checksum that is calculated usingthe node referenced by the address pointer. This arrangement means thatthe checksum is stored separately from the node from which it iscalculated. By storing the checksum of each node in its parent nodepointer and not in the node itself, every node in the tree contains thechecksums for all its children. This is illustrated in further detailwith respect to the examples of FIGS. 3A-3D. By doing this, each tree isautomatically self-validating, and it is always possible to detectinconsistencies with read operations.

If at block 729 data services are required, the next block 719 goes toblock 730 on FIG. 7B. Data services include, but are not limited to,compression, encryption, deduplication (which must be done in thatorder), snapshotting and cloning.

Compression is the process of reducing the data size of a data block(referred to interchangeably with leaf node or data node), typically byexploiting redundancies in the data block itself. Many differentcompression types are used by ZFS. When compression is enabled, lessstorage can be allocated for each data block. The following compressionalgorithms are available. LZ4—an algorithm added after feature flagswere created. It is significantly superior to LZJB. LZJB is the originaldefault compression algorithm) for ZFS. It was created to satisfy thedesire for a compression algorithm suitable for use in file systems.Specifically, that it provides fair compression, has a high compressionspeed, has a high decompression speed and detects incompressible datadetection quickly. GZIP (1 through 9 implemented in the classicLempel-Ziv implementation. It provides high compression, but it oftenmakes IO CPU-bound. ZLE (Zero Length Encoding)—a very simple algorithmthat only compresses zeroes. In each of these cases there is a trade-offof compression ratio to the amount of latency involved in compressingand decompressing the data block. Typically—the more compressed thedata—the longer it takes to compress and decompress it.

Encryption is the process of adding end-to-end security to data blocksby encoding them cryptographically with a key. Only users with a key candecrypt the data block. As used in the ZFS system, a ZFS pool cansupport a mix of encrypted and unencrypted ZFS data sets (file systemsand ZVOLs). Data encryption is completely transparent to applicationsand provides a very flexible system for securing data at rest, and itdoesn't require any application changes or qualification. FurthermoreZFS encryption randomly generates a local encryption key from apassphrase or an AES key and all keys are stored locally with theclient—not in the cloud object store 404 as traditional file systems do.Encryption is transparent to the application and storage to the cloudobject store 404 when turned on. ZFS makes it easy to encrypt data andmanage data encryption. You can have both encrypted and unencrypted filesystems in the same storage pool. You can also use different encryptionkeys for different systems, and you can manage encryption either locallyor remotely—although the randomly generated encryption key alwaysremains local. ZFS encryption is inheritable to descendent file systems.Data is encrypted using AES (Advanced Encryption Standard) with keylengths of 128, 192, and 256 in the CCM and GCM operation modes.

Deduplication is the process of recognizing that a data block to bestored in the file system is already stored on the file system as anexisting data block and pointing to that existing data block rather thanstoring the data block again. ZFS provides block-level deduplicationbecause this is the finest granularity that makes sense for ageneral-purpose storage system. Block-level deduplication also mapsnaturally to ZFS's 256-bit block checksums, which provide unique blocksignatures for all blocks in a storage pool as long as the checksumfunction is cryptographically strong (e.g. SHA256). Deduplication issynchronous and is performed as data blocks are sent to the cloud objectstore 404. If data blocks are not duplicated, enabling deduplicationwill add overhead without providing any benefit. If there are duplicatedata blocks, enabling deduplication will both save space and increaseperformance. The space savings are obvious; the performance improvementis due to the elimination of storage writes when storing duplicate data,plus the reduced memory footprint due to many applications sharing thesame pages of memory. Most storage environments contain a mix of datathat is mostly unique and data that is mostly replicated. ZFSdeduplication is per-dataset and can be enabled when it is likely tohelp.

A snapshot is a read-only copy of a file system or volume. A snapshot isa view of a filesystem as it was at a particular point in time in termsof changes to the tree image. ZFS's snapshots are useful in the same waythat some other filesystems's snapshots are: By doing a backup of asnapshot, you have a consistent, non-changing target for the backupprogram to work with. Snapshots can also be used to recover from recentmistakes, by copying the corrupted files from the snapshot. Snapshotscan be created almost instantly, and they initially consume noadditional disk space within the pool. However, as blocks (in this caseboth data and metadata) within the active dataset changes, the snapshotconsumes disk space by continuing to reference the old blocks, thuspreventing the storage space from being freed. The blocks containing theold data will only be freed if the snapshot is deleted. Taking asnapshot is a constant-time operation. The presence of snapshots doesn'tslow down any operations. Deleting snapshots takes time proportional tothe number of blocks that the delete will free and make available, andis very efficient. ZFS snapshots include the following features: theypersist across system reboots; the theoretical maximum number ofsnapshots is 2⁶⁴; they use no separate backing store; they consume diskspace directly from the same storage pool as the file system or volumefrom which they were created; recursive snapshots are created quickly asone atomic operation; and they are created together (all at once) or notcreated at all. The benefit of atomic snapshot operations is that thesnapshot data is always taken at one consistent time, even acrossdescendent file systems. Snapshots cannot be accessed directly, but theycan be cloned, backed up, rolled back to, and so on. Snapshots can beused to “roll back” to the point when the snapshot was taken

A clone is a writable volume or file system whose initial contents arethe same as the dataset from which it was created. In the ZFS systemclones are always created from snapshots. As with snapshots, creating aclone is nearly instantaneous and initially consumes no additional diskspace. In addition, you can snapshot a clone. Clones can only be createdfrom a snapshot. When a snapshot is cloned, an implicit dependency iscreated between the clone and snapshot. Even though the clone is createdsomewhere else in the dataset hierarchy, the original snapshot cannot bedestroyed as long as the clone exists. Clones do not inherit theproperties of the dataset from which it was created. A clone initiallyshares all its disk space with the original snapshot. As changes aremade to the clone, it uses more disk space. Clones are useful to branchoff and do development or troubleshooting—and can be promoted to replacethe live file system. Clones can also be used to duplicate a file systemon multiple machines.

Referring now back to FIG. 7B, flowchart 700-2 showing a method fordetermining and applying data services to data blocks. At decision block731 if compression is turned on or requested then the next block is 740.At decision block 732, if encryption is turned on or requested then thenext block is 750. At decision block 733, if deduplication is turned onor requested then the next block is 738. At decision block 734, if asnapshot is to be taken, the next block is 775. Block 735 returns toblock 716. FIG. 7B also illustrates the required ordering of anyrequested data services. Compression must be performed first followed byencryption, deduplication, and snapshotting and cloning. When datablocks are read, the reverse order must be employed.

As indicated by block 720, data objects and metadata corresponding tothe incremental modifications may be stored in the cloud object store404. In various embodiments, the cloud interface appliance 402 maycreate, read, forward, define, and/or otherwise specify data objects andmetadata corresponding to the incremental modifications. As indicated byblock 722, in some embodiments, the storage of the data objects andmetadata may be caused at least in part by the cloud interface device502 sending requests to the cloud interface daemon 604. As indicated byblock 724, the requests sent by the cloud interface device 502 to thecloud interface daemon 604 may be translated into put requests for thecloud client interface 608. The requests sent by the cloud interfacedevice 502 may be put requests in some embodiments; in otherembodiments, the cloud interface daemon 604 may translate the requestssent by the cloud interface device 502 into put requests.

As indicated by block 726, responsive to the requests sent by the cloudinterface device 502, the cloud interface daemon 604 may communicate,via the object protocol over the one or more networks, with the cloudobject store 404 to cause storage of the data objects and metadatacorresponding to the incremental modifications as new cloud objects. Asindicated by block 728, the cloud interface appliance 402 may update themapping 406 in view of the data objects and metadata corresponding tothe incremental modifications stored in the cloud object store 404.

Referring now to FIG. 7C depicting a flowchart 700-3 at block 742 fromblock 740. FIG. 7C depicts a flowchart of compressing the data blocks topreserve storage space. Compression is performed in the transactionalobject layer at the DMU 218 as shown in FIG. 2. Compression is typicallyturned on because it reduces the resources required to store (cloudobject store 404) and transmit data. Computational resources areconsumed in the DMU 218 in the compression process and, usually, in thereversal of the process (decompression). Data compression is subject toa space-time complexity trade-off. For instance, a compression schememay require intensive processing decompression fast enough to beconsumed as it is being decompressed. The design of data compressionschemes involves trade-offs among various factors, including the degreeof compression and the computational resources required to compress anddecompress the data. A compression type is received or retrieved atblock 742 by the DMU 218 to compress the data block. Many differenttypes of compression are supported by ZFS including, but not limited to,LZ4 LZJB, GZIP, and ZLE. At block 744 the data blocks is compressed bythe DMU 218 using the compression type. At decision block 746 it isdetermined if there are more data blocks from the tree hierarchy thatare to be written to the cloud object store 404 that need to becompressed. If so—then block 744 is repeated until the all blocks arecompressed by the DMU 218 using the compression type. Once all blocksare compressed, block 748 returns to block 732 of FIG. 7B.

FIG. 7D depicts a flow chart 700-4 of the process of encrypting the datablocks if encryption is requested or turned on. At block 752, thepassphrase or AES key is retrieved or provided to the DMU 218. ZFS usesa “wraparound” encryption key system that uses a passphrase or AES key,stored locally, to then randomly generate an encryption key to encryptthe data blocks as shown in block 754. The passphrase or AES key can bestored in any local storage including the ARC 224. Encryption does notof itself prevent the data blocks from being missapropriated, but deniesthe message content to the interceptor. In an encryption scheme, theintended data block, is encrypted using an encryption algorithm,generating ciphertext that can only be read if decrypted. For technicalreasons, an encryption scheme usually uses a pseudo-random encryptionkey generated by an algorithm. It is in principle possible to decryptthe message without possessing the key, but, for a well-designedencryption scheme, large computational resources and skill are required.ZFS Data blocks are encrypted using AES (Advanced Encryption Standard)with key lengths of 128, 192, and 256. The data blocks are encrypted atblock 756 using the randomly generated encryption key. At decision block758, it is determined if more data blocks need to be encrypted, and ifthere are, they are encrypted at block 756 until there are no more datablocks to encrypt. Then blocks 759 returns to block 733 of FIG. 7B todetermine if the data blocks need more data service processing.

FIG. 7E depicts the flowchart 700-5 to deduplicate data blocks in thecloud object store 404. Data block duplication a specialized datacompression technique for eliminating duplicate copies of repeatingdata. Data block deduplication is used to improve storage utilizationand can also be applied to network data transfers to reduce the numberof data blocks that must be sent to store in a COW memory. In thededuplication process, unique data blocks are identified and storedduring a process of analysis. As the analysis continues, other datablocks are compared to the stored copy and whenever a match occurs, theredundant data block is replaced with a small reference that points tothe stored data block. Given that the same data block pattern may occurdozens, hundreds, or even thousands of times the number of data blocksthat must be stored or transferred is reduced substantially usingdeduplication. This type of deduplication is different from thatperformed by standard file-compression discussed for FIG. 7C. Thatcompression identifies short repeated substrings inside individual datablocks, the intent of storage-based data deduplication is to inspectlarge volumes of data and identify entire data blocks that areidentical, in order to store only one copy of it. Consider, for examplea typical email system might contain 100 instances of the same 1 MB(megabyte) file attachment and if all 100 instances of the attachmentare saved, it would require 100 MB storage space. With datadeduplication, only one instance of the attachment is actually stored;the subsequent instances are referenced back to the saved copy fordeduplication ratio of roughly 100 to 1. Thus, data block deduplicationcan reduce the required storage place in the cloud object store 404 andreduce the pressure on the network transferring the data blocks to thecloud object store 404.

In FIG. 7D at block 762 the first process is to generate a name for thedata block using a name generation protocol. In ZFS that includes usinga checksum algorithm such as the SHA256. When the checksum algorithm isperformed on the data block, it generates a checksum that is unique tothe content of the data block. Thus—if any other data block has exactlythe same content—the checksum, using that same algorithm or namingprotocol, would be exactly the same. And that is the key to ZFS datadeduplication. At the decision block 764 it is determined if there is anexisting data block with the same name. This can be done in a multitudeof ways. One is to keep a local table of existing names. That, however,would limit data block deduplication to deduplicating only data blocksoriginating locally. The table of existing names can be stored on theobject store 404. The table on objet store 404 could be stored in theclient local data pool or globally and available to all clients. Wherethe data is stored at the object store will affect the amount of datablock compression that can be achieved through data block deduplication.For instance—if the table with existing names is global, only one copyof the 1 MB file discussed above would need to be stored on the cloudobject store 404 even if there were multiple clients using the cloudobject store 404. This would be the case even if the 1 MB attachment hadgone “viral” through email and ending up as an attachment to thousandsof emails. With a global existing name table on the cloud object store404—that 1 MB file would be stored only once on the cloud object store404—but referenced by perhaps 1000s of metadata blocks pointing to it.To that end at block 766 the data block with the same name as theexisting name ignored when doing further storage computations with thedata blocks and the pointer to the existing block is used to createmetadata blocks to generate the tree at block 768 according to themethod of blocks 716 and 718 in FIG. 7A. Decision block 770 causes arepeat of the process for as many data blocks in the tree by returningto block 762. At decision block 773 if a snapshot has been requested theat block 774 the next block is 775 in FIG. 7F. If a snapshot has notbeen requested the next block at 772 is block 720 in FIG. 7A.

FIG. 7F depicts the flowchart 700-6 of a method for ZFS snapshotting andcloning to a cloud object store 404. ZFS snapshots are the essential andeffortless backup mechanism for ZFS systems. ZFS snapshots are a“picture” of the ZFS tree hierarchy at moment in the tree's “life.” Asdiscussed in FIG. 3F, time, in terms of a snapshot, is based on thebirth time of the root block (used interchangeably with root node anduberblock) which is expressed in terms of which progression the rootblock was created in. A progression occurs each time a data block isgenerated to be stored. UItimately—the complete tree hierarchy isgenerated before taking a snapshot as described in FIGS. 3A-3D andblocks 716 and 718 of FIG. 7A. A reference to the root block is storedas depicted in block 780. The root block and all blocks active at thatprogression are part of the snapshot in that the snapshot referencepoints to the root block and the root block points to all blocks througheach lower level block, but they are not stored as duplicate blocks.Rather, as shown in block 782 all of the blocks in the tree thataccessible from the tree root block are marked as “do not free,” whichdesignates them as “Read Only” in ZFS syntax. In the normal progressionof the ZFS storage COW system, once a block is not active—in otherwords—it is not referenced by any higher level block—it is made free forother storage needs. The snapshot blocks must be kept intact for asnapshot to be useful at all. When a block is referenced through a rootblock referenced by a snapshot—the block cannot be “freed” until thesnapshot is deleted. This makes it possible to back up to the point thesnapshot was made until the snapshot is deleted. Snapshots can be storedin the cloud object store 404, ARC 222, L2ARC 222-3, or any other localstorage such as system storage 228. Using the ZFS file system, snapshotscan be requested at particular instances in the progression of thehierarchical tree and they can be automatically generated at aparticular periodic point in time. Old snapshots are not automaticallydeleted when new snapshots are created—so even after a new snapshot iscreated—a backup can occur to a previous snapshot as long as it has notbeen deleted. Thus—snapshots enable incremental backups—since the entirefile system does not have to copied indeed the entire backup is alreadyin the cloud object store 404 as pointed to by the root block referencedby the snapshot. The root block pointed to by the snapshot referencebecomes the active root block and all subsequent root blocks an blockcreated at a birth time after the snapshot can be freed for otherstorage use.

At decision block 784 that determines if a clone has been requested. Ifa clone has not been requested, at block 799 the snapshot process isover and block 720 in FIG. 7A is next. If a clone has been requested—itis made from the snapshot and as shown in block 786, a clone referencepoints to the same root block that the snapshot points to. A clone isalways generated from a snapshot such that a snapshot cannot be deleteduntil the clone is deleted as shown in block 788. Clones are used for avariety of purposes—for multiple development purposes from the same datastorage set, to instantiate a new virtual machine, to troubleshootissues, etc. To that end, clones must be made able to be COW whenaccessed from the clone reference. Clones are different than snapshotsin that respect—since no COW can occur from a snapshot. Clones need noextra storage capability at generation time—but as progressions are madeon the clone tree—clones will use more storage. Snapshots can be madefrom clones just as they can from active trees and for all the samereasons. Clones do not inherit the properties of the root block data.Clones can be ultimately be promoted to be the active tree as well. Atblock 799, clone generation is done and block 720 in FIG. 7A is next.

FIG. 8 is a high-level diagram illustrating an example of the cloudinterface appliance 402 handling incremental modifications, inaccordance with certain embodiments of the present disclosure. Certainembodiments may provide a very efficient incremental-always (also knownas incremental forever) backup capability, in addition to the ability torestore optimally (without applying multiple incremental updates) fromthe cloud. Traditional backup methods in the industry involve pushingcopies into the cloud. But certain embodiments according to the presentdisclosure allow for a cloud-based, copy-on-write file system where onlynew data objects are written to cloud storage. Cloud storage where onlynew and modified data needs to be sent provides an extremely effectivesolution for backing up and restoring from cloud providers. Old dataobjects need not be modified. These embodiments, along with theconsistency model using transaction groups, facilitates cloud storagewith incremental forever backup where consistency can always beconfirmed.

For illustration, FIG. 8 depicts the cloud interface appliance 402 ashaving created an initial backup (e.g., cloud data objects 414corresponding to backup logical tree 300A) of the local instance of thelogical tree 300. In some embodiments, the cloud interface appliance 402may utilize an active tree image 301 or a full copy to create thebackup. After a full backup of the local instance of the logical tree300 is initially created, a number of modifications 303 may be made tothe logical tree 300 with various transactions. In FIG. 8, the storingof the data objects and metadata corresponding to the incrementalmodification 303 is illustrated. The incremental modification 303 isdepicted, by way of example, with new leaf nodes 324, 326; new non-leafnodes 328, 330, 332, 334; and a new root node 336.

The cloud interface appliance 402 may be configured to createincremental backups, potentially indefinitely. To that end, certainembodiments of the cloud interface appliance 402 may utilize a snapshotCOW process (e.g., as disclosed above with respect to FIGS. 3E and 7F).In the example depicted in FIG. 8, may utilize an image 304 to createthe incremental modification. In some embodiments, the image 304 maycorrespond to an active tree image; in some embodiments, the image 304may correspond to a snapshot. With the image 304, the cloud interfaceappliance 402 may cause storage of an incremental modification 303A,which may correspond to cloud storage objects 414A. In accordance withthe cloud-based COW process, new cloud objects 414A are allocated forthe data objects and metadata corresponding to the incrementalmodification 303, with the cloud-based instantiation of the incrementalmodification 303 indicated as incremental modification 303A. The newroot node can then be made the root of the modified logical tree 300A-1with the storing operation to finalize the state of the modified logicaltree 300A-1. The modified logical tree 300A-1 may correspond to thelogical tree 300A modified by the incremental modification 303A.

With the storing of the incremental modification 303A, blocks of thelogical tree 300A (saved as cloud data objects 414) containing activedata may not be overwritten in place. The new cloud data objects 414Amay be allocated, and modified/new data and metadata may be written tothe cloud data objects 414A. The previous version of data may beretained, allowing a snapshot version of the logical tree 300A to bemaintained. In some embodiments, any unchanged data may be shared amongthe modified logical tree 300A-1 and its associated snapshots.

Thus, utilizing snapshots, such as the exemplary snapshot 301, anduberblocks, such as those corresponding to roots 336, 336-1, the cloudobject store 404 may be updated only with respect to a subset of nodesthat changed since the last snapshot. And that update of the incremental303A may hook in to the version of tree 300A in the cloud via the rootnode so that any part of the whole modified logical tree 300A-1 may beaccessed.

By sending incremental snapshots into the cloud date store 404, metadatathat can be retained inside the data instance which allows traversal ofthe tree and get a picture of the tree at time of snapshot. At the sametime, this allows for a very condensed representation of data that onlyretains data of blocks referenced that are specified/requested as to beretained in by the snapshot, such that only a minimal amount of dataneed be stored in the tree to retain point-in-time images of the tree.Every incremental may be merged with the tree 300A previously stored inthe cloud date store 404 so that, after each incremental is sent, sothat a full, current representation of data always exists. Each mergingof an additional incremental with the tree 300A may results in singledata instance such that full backup operations are unnecessary.

Although unnecessary from a performance perspective, in someembodiments, full back-ups may be made periodically to avoid having togather up a high number of incrementals if that is not desired by aclient. With some implementations, multiple versions of full backups maybe retained, if desired. Thus, certain embodiments provide for completecontrol of intervals, snapshots, and backups. Advantageously, certainembodiments may be configured to dynamically self-adjust incrementalintervals. For example, some embodiments may initially operate accordingto a first interval. In various instances, an interval could be withevery write operation to the local tree, every half hour, every day,etc. When a churn rate (e.g., a metric monitored by the hybrid cloudstorage system 400 that indicates rates of changes the local tree)exceeds (or decrease to) a certain churn threshold, the hybrid cloudstorage system 400 may automatically transition to a different interval.For example, if the churn rate exceeds a first churn threshold, thehybrid cloud storage system 400 may automatically transition to agreater interval of time for the incremental interval. Likewise, if thechurn rate decreases to the first churn threshold or another churnthreshold, the hybrid cloud storage system 400 may automaticallytransition to a lesser interval of time for the incremental interval.Certain embodiments may employ multiple churn thresholds to throttleincremental frequencies per a gradated scheme. Similarly, certainembodiments may dynamically self-adjust full backup intervals based atleast in part on such churn thresholds and/or incremental size and/ornumber thresholds, which could be client-defined in some instances.

One of the chief problems with writing an object into a cloud andsubsequently reading the object from the cloud is that integrity of theobject read is not guaranteed. There is a risk of degradation of dataincumbent with the cloud storage (e.g., storage loss, transmissionfailure, bitrot, etc.). Furthermore, with the involvement of multipleversions of objects, there is a risk of reading an undesired version ofan object. For example, a previous version of the desired object may beread, such as the most recent version in an instance where an update ofthe object is incomplete.

Traditional object store architectures rely on copies of data wherein aquorum (i.e., 2 copies) is satisfied initially and additional copies(e.g., a third copy) are updated asynchronously. This presents thepossibility that a client could receive a copy of data before all copiesbeen updated, and consequently get an inconsistent view of the data. Atypical solution for object store consistency is to ensure that allcopies are made prior to the data being available, however ensuringconsistency with that solution is typically not realistic or attainable.An object-based approach where the checksum of the object is stored inthe object's metadata alongside the object or in a reference databaseelsewhere would allow validation of the object content but would notvalidate the correct version of the object. Moreover, solutions thatmight use object versioning and check from a single location wouldsomewhat defeat the purpose and intent of cloud storage.

However, certain embodiments according to the present disclosure mayprovide consistency model that can ensure guaranteed integrity in thecloud and that can ensure always-consistent semantics from an eventuallyconsistent object model. For example, certain embodiments may providefault isolation and consistency through the use of logical treesdisclosed herein. With all transactional write operations to the cloudobject stores, the checksums in the metadata of the self-describingMerkle tree may be updated. As described above, the storing of checksumsseparately from the node from which the checksums are calculated ensuresthat each tree is automatically self-validating. At every tree layer,the node below is referenced by a node pointer that includes a checksum.So, when an object is read out of the cloud, it may be determinedwhether that object is correct.

FIG. 9 is a block diagram that illustrates an example method 900directed to certain features of the hybrid cloud storage system 400 thatensure guaranteed integrity in the cloud and always-consistent semanticsfrom an eventually consistent object model, in accordance with certainembodiments of the present disclosure. According to certain embodiments,the method 900 may begin as indicated by block 902. However, asclarified above, teachings of the present disclosure may be implementedin a variety of configurations such that the order of certain steps ofmethods disclosed herein may be shuffled or combined in any suitablemanner and may depend on the implementation chosen. Moreover, while thefollowing steps may be separated for the sake of description, it shouldbe understood that certain steps may be performed simultaneously orsubstantially simultaneously.

As indicated by block 902, a POSIX-compliant request to perform one ormore particular operations (i.e., one or more transactions) may bereceived from the application 202. Such an operation may correspond toreading or otherwise accessing data. As indicated by block 904, thePOSIX-compliant request may be forwarded from the operating system, viathe system call interface 208, to the DMU 218. As indicated by block906, the DMU 218 may translate requests to perform operations on dataobjects directly to requests to perform one or more read operations(i.e., one or more I/O requests) directed to the cloud object store 404.As indicated by block 908, the SPA may receive the I/O request(s) fromthe DMU 218. Responsive to the request(s), the SPA may initiate readingof one or more data objects from the cloud object store 404.

As indicated by block 910, the cloud interface appliance 402 may receivethe I/O request(s) and may send corresponding cloud interface request(s)to the cloud object store 404. In some embodiments, the cloud interfaceappliance 402 may translate the I/O requests into corresponding objectinterface requests. The cloud interface device 502 may coordinate theobject interface requests using the mapping 406 of cloud storage objects414 in the cloud object store 404. For example, the cloud interfaceappliance 402 may identify and request all or a portion of a file storedas cloud data objects in accordance with a logical tree.

As indicated by block 912, the cloud interface appliance 402 may receivedata object(s) responsive to the object interface requests. As indicatedby block 914, the data object(s) may be checked with a checksum(s) fromparent node(s) in the logical tree. In various embodiments, the checkingmay be performed by the cloud interface appliance 402 and/or the I/Opipeline 224.

FIG. 10 is a high-level diagram illustrating an example of the cloudinterface appliance 402 handling the checking, in accordance withcertain embodiments of the present disclosure. Again, in otherembodiments, another component of the system 400 may perform thechecking. But, in FIG. 10, the cloud interface appliance 402 is depictedas having accessed the cloud object store 404 to read data storedaccording to the logical tree 300A-1. In the example depicted, the cloudstorage appliance 402 is illustrated as having accessed cloud objects414A corresponding to the logical tree 300A-1. The cloud storageappliance 402 accessed the leaf node 324-2 by way of the addresses ofthe non-leaf nodes 326-2, 334-2, and the root node 336-2.

As described herein, when reading a node out of the logical tree, thenode is read using a pointer of a node in higher level in the logicaltree. That pointer includes a checksum for the data that is expected tobe read so that, when data is pulled from the cloud, the data may bechecked with the checksum. The actual checksum of the data may becalculated and compared to the expected checksum already obtained by thecloud interface appliance 402 and/or the I/O pipeline 224. In theexample depicted, the non-leaf node 326-2 includes the checksum for leafnode 324-2.

In some embodiments, a checksum may be received from the cloud datastore with one object and the data to be checked with the checksum isreceived with a different object. The checksum may be received from thecloud data store with the separate object prior to receiving thedifferent object with the data to be checked, according to someembodiments. Some embodiments may employ an iterative object interfacerequest process such that, in response to a particular object interfacerequest, the object with the checksum is received, and, in response to asubsequent object interface request, the object with the data to bechecked is received. Further, with some embodiments, the subsequentobject interface request may be made using addressing information thatwas received with the particular object interface request and points tothe object with the data to be checked. With some embodiments, insteadof an iterative process, a plurality of objects is received from thecloud data store, after which the checking may be performed on theplurality of objects. The integrity of the logical tree can be quicklychecked by traversing the logical tree and calculating checksums ofchild nodes at each level based on parent nodes. In alternativeembodiments, the cloud interface appliance 402 and/or the I/O pipeline224 may obtain the checksum prior to initiating the read operationsand/or may obtain the checksum from another source. For example, in suchalternative embodiments, the checksum may be retained in the mapping406, a snapshot of the logical tree, and/or a local data store.

Referring again to FIG. 9, as indicated by block 916, it may bedetermined whether to validate the data object(s) by the checksum(s)from parent node(s) in the logical tree with the actual checksum(s) ofthe data object(s). As indicated by block 918, in the case of the dataobject(s) being validated, the reading and/or further processingoperations of the system 400 may proceed, as the data has beendetermined to be not corrupted and not the incorrect version. Asnecessary, checking may proceed with additional objects until allobjects are check summed and validated by parent object pointermetadata.

In the case of mismatch of actual and expected checksums of the dataobject(s), an error condition may be identified. This is the caseillustrated in FIG. 10, where the actual checksum of the object D′(which corresponds to leaf node 324-2) does not match the checksumspecified by the parent node 326-2 (which corresponds to object C′).With an error condition, process flow of FIG. 9 may proceed to block920. The mismatch of actual and expected checksums may correspond tosituations of getting the wrong version of data that is not up to date,degradation of data due to cloud storage failure, bitrot, and/ortransmission loss, etc. As indicated by block 920, remediation may beinitiated. In some embodiments, the remediation may include reissuingone or more cloud interface requests to the cloud object store 404.Responsive to each reissued request, process flow may return to block912, where the cloud interface appliance 402 may receive one or moredata objects and another iteration of the checking process may proceed.

The reissuance of the one or more cloud interface requests maycorrespond to requests that the cloud object store 404 try harder tofind the correct version of the object requested. With some embodiments,the cloud interface appliance 402 may iteratively go through cloudnodes/devices until the correct version is retrieved. Some embodimentsmay iteratively check snapshots of previous versions of tree portionsthat may be stored in the cloud object store 404. Some embodiments mayemploy a threshold such that the cloud interface appliance 402 mayproceed to other remediation measures after the threshold has beensatisfied. The threshold may correspond to a number of reissuances ofthe one or more cloud interface requests. Alternatively or additionally,the threshold may correspond to an limit on the extent of the search ofnodes and/or snapshots. For example, the threshold may govern how manynodes and/or how many snapshots are to be searched before the cloudinterface appliance 402 turns to a different remediation measure.

Another remediation measure that the cloud interface appliance 402 mayemploy is the requesting the correct version of the data from anothercloud object store in multiple-cloud implementations, as indicated byblock 924. Certain embodiments of multiple-cloud storage are describedfurther herein. A record of the second cloud object store may beinspected to determine whether a copy of the data had been stored in thesecond cloud object store. In some embodiments, the record couldcorrespond to another mapping 406 specific to the second cloud objectstore. Having determined that a copy of the data had been stored in thesecond cloud object store, the cloud interface appliance 402 mayinitiate one or more object interface requests to the second cloudobject store in order to retrieve the desired data.

With such embodiments, the request for the object of interest may bemade to the second cloud object store after the threshold of reissuedrequests to the cloud object store 404 has been satisfied withoutsuccessfully received the correct version. Alternatively, someimplementations could resort to the second cloud object store as a firstdefault, rather than reissuing requests to the cloud object store 404.In any case, responsive to the request to the second cloud object store,process flow may return to block 912, where the cloud interfaceappliance 402 may receive one or more data objects and another iterationof the checking process may proceed. In the case of the correct databeing retrieved from the second cloud object store and being validated,the reading and/or further processing operations of the system 400 mayproceed, as indicated by block 918. Additionally, having receivedcorrect data from the second cloud object store, the system 400 maywrite the correct data to the cloud object store 404, as indicated byblock 926. The writing of the correct data may be effected with a COWprocedure to apply an incremental modification, as disclosed withrespect to FIG. 7.

In some embodiments, if the correct data is not retrievable through theremediation processes, an error message and a most recent version of thedata may be returned, as indicated by block 928. With some instances,the most recent version of the data may not be retrievable. For example,metadata that should point to the data may be corrupted such that themost recent version of the data cannot be referenced and retrieved. Inthose instances, an error message may be returned without a most recentversion of the data. However, when the most recent version of the datais retrievable, it may be returned, as well. Accordingly, with anend-to-end checksum model, certain embodiments may cover end-to-endtraversal of the data from the client into the cloud and back again.Certain embodiments may not only provide for check summing of an entiretree and detecting errors, but also the ability to correct portions ofthe tree through resilvering and data scrubbing.

FIG. 11 is a diagram of a simplified example further illustratingfeatures of a hybrid cloud storage system 400-2, in accordance withcertain embodiments of the present disclosure. The hybrid cloud storagesystem 400-2 illustrates how a hybrid storage pool 1100 is at leastpartially formed of the system storage pool 416 and the virtual storagepool 602-1. Flows for read operations and write operations areillustrated for each of the system storage pool 416 and the virtualstorage pool 602-1.

In some embodiments, the ARC 222 may include the ARC 222-2. Readoperations of the system storage pool 416 may be facilitated by the ARC222-2 and system storage blocks 228-1. As indicated, certain embodimentsof the ARC 222-2 may be implemented with DRAM. As is also indicated,certain embodiments of the system storage blocks 228-1 may have aSAS/SATA-based implementation. With some embodiments, read operations ofthe system storage pool 416 may be further facilitated by a L2ARC device222-3 that may extend the cache size. With some embodiments, the ARC 222may be referenced as inclusive of the L2ARC device 222-3. As indicatedin FIG. 11, certain embodiments of the L2ARC device 222-3 may have aSSD-based implementation. Write operations of the system storage pool416 may be facilitated by the system storage blocks 228-1 and the intentlog 214-3. As indicated, certain embodiments of the intent log 214-3 mayhave a SSD-based implementation.

To the local system, the virtual storage pool 602-1 may appear andbehave as a logical disk. Similar to the system storage pool 416, readoperations of the virtual storage pool 602-1 may be facilitated by theARC 222-4 and the cloud storage objects blocks 414-1. With someembodiments, the ARC 222 may be referenced as inclusive of the ARC222-4, even though the ARC 222-4 may be implemented with one or moreseparate devices in some embodiments. As indicated in FIG. 11, certainembodiments of the ARC 222-4 may be implemented with DRAM. In someembodiments, the ARC 222-4 may be the same cache as the ARC 222-2; inother embodiments, the ARC 222-4 may be a separate cache of the virtualstorage pool 602-1 that is distinct from the ARC 222-2. As indicated,certain embodiments facilitating the cloud storage objects blocks 414-1may have an HTTP-based implementation.

With some embodiments, read operations of the virtual storage pool 602-1may be further facilitated by a L2ARC device 222-5. With someembodiments, the ARC 222 may be referenced as inclusive of the L2ARCdevice 222-5. In some embodiments, the LSARC device 222-5 may be thesame cache as the L2ARC device 222-3; in other embodiments, the LSARCdevice 222-5 may be a separate cache device of the virtual storage pool602-1 that is distinct from the L2ARC device 222-3. As indicated,certain embodiments of the L2ARC device 222-5 may have a SSD-based or aHDD-based implementation.

Write operations of the virtual storage pool 602-1 to the cloud storageobjects blocks 414-1 may be facilitated by the intent log 214-4. In someembodiments, the intent log 214-4 may be the same as the intent log214-3; in other embodiments, intent log 214-4 may be a separate anddistinct from the intent log 214-3. As indicated, certain embodiments ofthe intent log 214-4 may have a SSD-based or a HDD-based implementation.

The transition to cloud storage provides several advantages (e.g., cost,scale, and geographic locality), but, when used conventionally, cloudstorage comes with some limitations. When application clients arelocated on premise and not co-resident in the cloud, latency isfrequently a significant concern with conventional technologies.However, the hybrid cloud storage system 400 can eliminate that concern.Certain embodiments of the hybrid cloud storage system 400 may providemirroring features to facilitate performance, migration, andavailability.

With the hybrid cloud storage system 400, the differential betweenlatencies of read operations of the system storage pool 416 and readoperations of the virtual storage pool 602-1 may be minimized. The ARCsand L2ARC devices of the two pools could be local implementations insome embodiments. And the latencies at the ARCs and L2ARC devices of thetwo pools may be equivalent or substantially equivalent. For example,typical latencies for read operations of the ARCs may be 0.01 ms orless, and typical latencies for the L2ARCs may be 0.10 ms or less.Latencies of read operations from the cloud storage objects blocks 414-1may be higher, but the hybrid cloud storage system 400 may intelligentlymanage both pools to minimize the higher latencies.

Certain embodiments may provide for low-latency, direct cloud accesswith file system semantics. Certain embodiments may facilitate runningfrom cloud storage while preserving application semantics. Certainembodiments may enable local storage read performance while retainingreplicated copies of all data in the cloud. In order to provide thosefeatures, certain embodiments may utilize on premise caching devices andleverage the hybrid storage pool caching algorithms. By providing cloudstorage with effective caching and file system semantics, cloud storagemay be employed for more than backup and restore. The hybrid storagesystem may use cloud storage “live.” Stated otherwise, through theintelligent use of on premise caching devices, the benefit of fullperformance may be provided to the local system without having to keepfull or multiple copies locally.

FIG. 12 is a diagram of a simplified example further illustratingfeatures of a hybrid cloud storage system 400-2, in accordance withcertain embodiments of the present disclosure. With effective cachingdevices and algorithms, the hybrid cloud storage system 400-2 may cacheat the block level and not at the file level, so accesses to largeobjects (e.g., a large database) do not require caching of the entireobject. In various embodiments, the example depicted may correspond toone or a combination of the virtual storage pool 602-1 and the systemstorage pool 416.

The hybrid cloud storage system 400-2 may employ adaptive I/O staging tocapture most of the objects needed for system operations. The hybridcloud storage system 400-2 may configure a plurality of cache devices toprovide adaptive I/O staging. In the example depicted, adaptive I/Ostaging is implemented with an ARC 222-5. Yet, in various embodiments,the hybrid cloud storage system 400-2 may be configured to use aplurality of ARCs 222 and/or L2ARCs 222 to provide adaptive I/O staging.While the following description uses an ARC 222 as an example, it shouldbe understood that various embodiments may use a plurality of ARCs 222,as well as one or more L2ARCs 222, to provide the disclosed features.

In some embodiments, the ARC 222-5 may be self-tuning such that the ARC222-5 may adjust based on the I/O workload. By way of example, inembodiments where the hybrid cloud storage system 400-2 uses cloudstorage in a live mode, not merely for back-up and migration, the ARC222-5 may provide caching algorithms that stage objects according to aprecedence order. That precedence order for caching may correspond tomost recently used (MRU) objects, most frequently used (MFU) objects,least frequently used (LFU) objects, and least recently used (LRU)objects. With each I/O operation, the ARC 222-5 may determine whetherself-adjustment of staged data objects is necessary. Note that, incertain embodiments, the L2ARC 225-5 may work in conjunction with ARC222-5 to facilitate one or more of the stages. By way of example, theL2ARC 225-5, which may have a higher latency than the ARC 222-5, may beused for one or more of the lower ranked stages, such as the LRU and/orLFU stages. In some embodiments, another component of the hybrid cloudstorage system 400-2 may cause the caching in accordance with theseembodiments. By way of example, the cloud storage appliance 402 maycoordinate the caching and servicing of read and write requests.Further, the cloud storage appliance 402 may include the ARC(s) 222-5,L2ARC(s) 222-5, and/or intent log 214-4 according to some embodiments.

With each I/O operation, the ARC 222-5 may adjust the staging of one ormore objects previously staged to some extent. At a minimum, theadjustment may include updating tracking of accesses of at least oneobject. The adjustment may include demotion to a lower stage, eviction,or promotion to a higher stage. The transition criteria for promotionand demotion may be different for each transition from a current stageto another stage or to eviction. As disclosed herein, the ARC 222-5 mayhave the ability to evict memory buffers from the cache as a result ofmemory pressure to maintain a high throughput and/or to meet usagethresholds.

With a given I/O operation, if the one or more objects corresponding tothe I/O operation had not already been staged as MRU objects, then theone or more objects may be newly staged as MRU objects. Yet, if the oneor more objects corresponding to the I/O operation are already be stagedas MRU objects, the ARC 222-5 may apply transition criteria to the oneor more objects to determine whether to transition the one or moreobjects to a different stage. If the transition criteria is not met, nochange in staging is necessary with the servicing of the I/O operation.

FIG. 13 is a block diagram that illustrates an example method 1300directed to certain features of the hybrid cloud storage system 400-3for cache management and cloud latency masking, in accordance withcertain embodiments of the present disclosure. According to certainembodiments, the method 1300 may begin as indicated by block 1302.However, as clarified above, certain steps of methods disclosed hereinmay be shuffled, combined, and/or performed simultaneously orsubstantially simultaneously in any suitable manner and may depend onthe implementation chosen.

As indicated by block 1302, POSIX-compliant request(s) to performparticular operation(s) (i.e., a transaction(s)) may be received fromthe application 202. Such an operation may correspond to reading,writing, and/or modifying data. As indicated by block 1304, thePOSIX-compliant request(s) may be forwarded from the operating system,via the system call interface 208, to the DMU 218. As indicated by block1306, the DMU 218 may translate requests to perform operations onobjects directly to requests to perform I/O operations directed to aphysical location within the cloud object store 404. The DMU 218 mayforward the I/O requests to the SPA.

As indicated by block 1308, the SPA may receive the I/O requests fromthe DMU 218. And, responsive to the requests, the SPA may initiateperforming I/O operations. As indicated by block 1310, in the case ofwrite operations, the SPA may initiate writing of objects to the cloudobject store 404 using a COW procedure. For example, the cloud interfaceappliance 402 may proceed with the COW procedure disclosed above (e.g.,in view of FIG. 7) with respect to the cloud object store 404. Asindicated by block 1312, in the case of read operations, the SPA mayinitiate reading of objects. The ARC 222-5 may be checked for the one ormore requested objects. In some embodiments, as indicated by block 1314,it may be determined whether one or more validated data objectscorresponding to the I/O request(s) exist in the ARC 222-5. This mayinclude the SPA first determining if one or more objects correspondingto the read request(s) is retrievable from the ARC 222-5. Then, if suchone or more objects are retrievable, the object(s) may be checked withone or more checksums from one or more parent nodes in the logical tree.In various embodiments, the checking may be performed by the ARC 222-5,the cloud storage appliance 402, and/or the I/O pipeline 224. Asindicated by block 1316, in the case of the data object(s) beingvalidated (or, in some embodiments not employing validation, in thesimple case of a hit), the reading and/or further processing operationsof the system 400 may proceed.

In some embodiments, as indicated by block 1318, in the case of the dataobject(s) not being validated (or, in some embodiments not employingvalidation, in the simple case of no hit), the SPA may initiate readingof one or more objects from the local storage 228. With someembodiments, a pointer to the one or more objects may be cached and usedto read the one or more objects from the local storage 228. Someimplementations may not check local storage 228 for the one or moreobjects if such a pointer is not cached. If the one or more objects areretrievable, the one or more objects may be checked with one or morechecksums from one or more parent nodes in the logical tree. Again, invarious embodiments, the checking may be performed by the ARC 222-5, thecloud storage appliance 402, and/or the I/O pipeline 224. As indicate byblock 1320, in the case of the object(s) being validated, the processflow may transition to block 1316, and the reading and/or furtherprocessing operations of the system 400 may proceed.

As indicated by block 1320, in the case of the data object(s) not beingvalidated (or, in some embodiments not employing validation, in thesimple case of no hit), the process flow may transition to block 1322.As indicated by block 1322, the SPA may initiate reading of the one ormore data objects from the cloud object store 404. In variousembodiments, the imitating of the reading may be performed by one or acombination of the DMU 218, the ARC 222-5, the I/O pipeline 224, and/orthe cloud interface appliance 402. The reading of the one or moreobjects from the cloud object store 404 may include steps previouslydisclosed herein, for example, with regard to FIG. 9. Such steps mayinclude one or a combination of issuing I/O request(s) to the cloudinterface appliance 402, sending corresponding cloud interfacerequest(s) to the cloud object store 404 using the mapping 406 of cloudstorage objects 414, receiving data object(s) responsive to the objectinterface requests, and the like, which are detailed above. As indicatedby block 1324, it may be determined whether a validated object has beenretrieved from the cloud object store 404. Again, this may involve stepspreviously disclosed herein, for example, with regard to FIG. 9, whereit is determined whether to validate the data object(s) by thechecksum(s) from parent node(s) in the logical tree.

In the case of mismatch of actual and expected checksums of the dataobject(s), the process flow may transition to block 1326, whereremediation processes may be initiated. This may involve stepspreviously disclosed herein, for example, with regard to FIG. 9, whereremediation processes are disclosed. However, in the case of the databeing validated, the process flow may transition to block 1316, and thereading and/or further processing operations of the system 400 mayproceed.

As indicated by block 1328, having retrieved the one or more objects,the cache staging may be adjusted. In certain instances, the adjustmentof cache staging may include newly caching one or more objects as MRUobjects, as indicated by block 1330. If the one or more objectscorresponding to a given I/O operation had not already been staged asMRU objects, then the one or more objects may be newly staged as MRUobjects.

Yet, in certain instances, when the one or more objects corresponding tothe I/O operation are already be staged as MRU, MFU, LFU, or LRUobjects, the ARC 222-5 may apply transition criteria to the one or moreobjects to determine whether to transition the one or more objects to adifferent stage, as indicated by block 1332. However, if the transitioncriteria is not met, a change in staging may not be necessary with theservicing of the I/O operation.

In some embodiments, the staging of objects may be at least partially afunction of recency of access of the objects. As indicated by block1334, in some embodiments, the adjustment of cache staging may includeupdating one or more recency attributes. The ARC 222-5 may define arecency attribute for the one or more new objects in order to trackrecency of access of the one or more objects. The recency attribute maycorrespond to a time parameter that indicates a last access timecorresponding to one or more objects (e.g., by absolute time, systemtime, time differential, etc.) and/or a sequential parameter thatindicates an access count corresponding to the one or more objectsagainst which recency attributes of other objects may be compared.

In various embodiments, the transition criteria may include one or morerecency thresholds defined in order for objects to qualify fortransition from current stages. For example, the ARC 222-5 may determineif the one or more objects should be transitioned to LFU or LRU stages(or eviction) based at least in part on the value of the recencyattribute assigned to the one or more objects. In some embodiments, therecency threshold may be a dynamic threshold, adjusted as a function ofrecency attributes defined for other objects in one or more stages. Forexample, the recency threshold may be a function of a lowest value ofany recency attribute defined for any objects already staged as MFUobjects, when the values of recency attributes defined for the stagedobjects are sorted in an ascending or descending order.

Additionally or alternatively, in some embodiments, the staging ofobjects may be at least partially a function of frequency of access ofthe objects. As indicated by block 1336, in some embodiments, theadjustment of cache staging may include updating one or more frequencyattributes. With the particular I/O operation, the ARC 222-5 mayincrement a frequency attribute defined for the one or more objects inorder to track the frequency of access of the one or more objects. Thefrequency attribute may indicate numbers of accesses over any suitabletime period, which could be an absolute time period, an activity-basedtime period (e.g., a user session, or time since a last amount of accessactivity that meets a minimum activity threshold), and/or the like.

In various embodiments, the transition criteria may include one or morefrequency thresholds defined in order for objects to qualify fortransition from current stages. For example, consequent to a change inthe value of the frequency attribute, the ARC 222-5 may determine if theone or more objects should be staged as MFU objects (or as objects inanother stage). Such a determination may be made based at least in parton comparing the updated frequency attribute to a frequency threshold.In some embodiments, the frequency threshold may be a dynamic threshold,adjusted as a function of frequency attributes defined for other stagedobjects (e.g., staged as MFU objects or objects in another stage). Forexample, the frequency threshold may be a function of a lowest value ofany frequency attribute defined for any objects already staged as MFUobjects, when the values of frequency attributes defined for the stagedobjects are sorted in an ascending or descending order.

As indicated by block 1338, additionally or alternatively, theadjustment of cache staging may include specifying one or more otherstaging attributes, according to some embodiments. A staging attributecould indicate an operation type. With some embodiments, the staging ofobjects may be at least partially a function of operation type. Forexample, the staging algorithm could employ discrimination of writeoperations versus read operations so that only objects accessed with aread operation may be staged as MFU objects. In such embodiments,objects referenced with write operations may be initially maintained asMRU objects and be thereafter subject to demotion according to LFUstaging criteria. Alternatively, in such embodiments, objects referencedwith write operations may be initially maintained as MRU objects and bethereafter subject to demotion according to LRU staging criteria andthen be subject to eviction, effectively skipping LFU staging. Asanother alternative, objects referenced with write operations may beinitially maintained as MRU objects and be thereafter subject toeviction, effectively skipping LFU staging and LRU staging. With suchalternatives, the ARC 222-5 discriminates write operations to commitcloud objects to the cloud object store 404 as being less likely neededfor subsequent read operations, hence allowing such potential operationsto incur cloud access latencies should the operations arise.

A staging attribute could indicate a data type. Additionally oralternatively, in some embodiments, the staging of objects may be atleast partially a function of data type. For example, some embodimentsmay accord a higher priority to metadata vis-à-vis data. That higherpriority may include retaining all metadata objects, and subjecting dataobjects to staging. Alternatively, that higher priority may includeapplying different criteria for staging transition (promotion and/ordemotion from current stages) to metadata objects versus data objects.For example, thresholds (e.g., recency, frequency, and/or the likethresholds) defined for data objects to qualify for demotion may belower (and, hence, more easily satisfied) than thresholds defined formetadata objects to qualify for demotion. Alternatively, otherembodiments may accord a higher priority to data vis-à-vis metadata.With some embodiments, a portion of the cloud objects may be defined asalways to cached regardless of frequency of use.

A staging attribute could indicate an operation characteristic.Additionally or alternatively, in some embodiments, the staging ofobjects may be at least partially a function of read operationcharacteristics. For example, some embodiments may accord a higherpriority to read operations having a size characteristic such that thesize of objects read satisfies a size threshold. Additionally oralternatively, a higher priority may be accorded to read operationshaving a sequential characteristic such that the sequence of objectsread satisfies a sequence threshold. Accordingly, a large, sequentialstreaming read operation may be given a higher priority for caching thansmaller, more isolated read operations. In that way, higher cloud-accesslatencies for the large, sequential streaming read operations areavoided.

Certain embodiments of the ARC 222-5 may employ different functions,transition criteria, and/or thresholds for each stage. In someembodiments, the ARC 222-5 may employ a staging scoring system. Someembodiments may score objects with a numerical expression, for example,a staging score. The staging scores may reflect qualifications of theobjects with respect to any suitable criteria, such as the transitioncriteria. For example, a given staging score for an object may becumulative of scoring according to criteria such as frequency of access,recency of access, operation type, data type, operation characteristics,object size, and/or the like. The given object could be scored withrespect to each criterion. For example, relatively greater values ofattributes such as frequency attributes, recency attributes, and/or likemay be accorded greater scores. Likewise, scores may be assigned in viewof the other criteria and priorities. The cumulative staging score forthe object may be used, along with the staging scores of other objectsstored in the cache, to rank the objects according to a precedenceorder. Again, the precedence order may be used to transition the objectsto different stages and/or toward eviction.

The ARC 222-5 may adapt the stages and the objects stored therein tosatisfy one or more cache usage and capacity constraints. For example,given a cache device capacity of, say, 1 TB of DRAM, the ARC 222-5 mayadapt the stages and the objects stored therein to maintain a maximumcache usage of 80%. In addition, some embodiments may adapt the stagesand the objects stored therein to satisfy one or more speed constraints.For example, the ARC 222-5 may monitor throughput to maintain anacceptable amount of access latencies (e.g., average access latencies)given both local accesses and cloud accesses in order to determinewhether more or less caching should be employed to satisfy one or morelatency tolerances. In view of such adaption constraints, adaption ofthe stages and the objects stored therein may include apply thedifferent functions and thresholds for each stage in order to sort theobjects in a precedence order. The precedence order may be utilized bythe ARC 222-5 to shift stored objects toward eviction in order to meetthe adaption constraints.

As indicated by block 1340, in some embodiments, the cache stagingadjustment may include transitioning to a different caching mode. Someembodiments may dynamically change modes of operation in order to loadbalance while meeting usage and latency constraints. An initial ordefault mode of operation may correspond to operating from the cloud ina live manner such that objects are accessed from cache first, then, ifnecessary, from the cloud. Some embodiments of the ARC 222-5 mayinitially (e.g., within a session or time period) cache all objects thatare accessed with I/O operations, and then transition to employingstaging as cache usage meets one or more thresholds. The transition tostaging may be incremental with one or more secondary modes ofoperation. For example, staging may be initially relegated to MRU andMFU staging, and then expanded to one or more of the other stages as oneor more cache usage thresholds (which may be preliminary to, and lowerthan, the maximum cache usage threshold) are met.

In view of cache usage approaching usage constraints and meeting one ormore usage thresholds, certain transition criteria may be appliedincrementally with one or more additional modes of operation. Forexample, objects corresponding to write operations may not be initiallydiscriminated against. Yet, as the cache usage approaches a usageconstraint, that discrimination criteria may be applied after one ormore usage thresholds are met.

As another example, as the cache usage further approaching a usageconstraint and meeting one or more usage thresholds, the hybrid cloudstorage system 400-2 may begin to make use of extended cache with one ormore L2ARC devices (which may correspond to one or more low-latencySides) for lower ranked stages (e.g., LRU and/or LFU stages). As stillanother example, as the cache usage further approaching a usageconstraint and meeting one or more usage thresholds, the hybrid cloudstorage system 400-2 may begin to make use of local storage 228 in orderto conform to latency tolerances with one or more tertiary modes ofoperation. By way of a more specific example, rather than evicting alarge, sequential streaming read operation without provisioning forfuture low-latency access, the extend of the operation and the frequencyof access of the corresponding objects may be sufficient to meet sizeand frequency thresholds such that the objects are to be transitioned tolocal storage 228. In this way, the hybrid cloud storage system 400-2may keep the objects available for local-latency read operations, whilefreeing up cache capacity for other low-latency access (which couldentail other large, sequential read operations) and avoiding payingcloud latencies should the large, sequential read operation be calledfor again. Such selective utilization of both local storage 228 andcloud storage at the object level may further facilitate masking ofcloud latencies while using cache a majority of the time and loadbalancing between cloud storage and local storage 228 to operate withinlatency tolerances. In various embodiments, this trifurcated storageadaptation can be initiated as a fallback operational mode or an initialdefault for certain types of operations have certain characteristics.

Accordingly, certain embodiments may alter caching modes and techniquesbased at least in part on the characteristics of object access. Certainembodiments may leverage caching features, cloud storage, and localstorage 228 to mask latencies for cloud-based operations. In suchembodiments, a majority of operations may be serviced from cache withcache hit rates typically exceeding 90% or more, which results in locallatencies most of the time. If any local object is missing or corrupt,the cloud copy of the object may be accessed. With some embodiments,reading from the cloud object store 404 may only be necessary when thereis no cache hit and the read request cannot be serviced from localstorage 228.

In some embodiments, in lieu of the ARC checking, the local storagechecking, and/or the cloud object store checking disclosed above, themapping 406 may be used to identify the location of the one or moreobjects of interest. As described above, the mapping 406 may include anobject directory and may maintain object states that are updated withevery I/O operation. Cloud object states may be kept in an index, atable, an index-organized table, and/or the like which may be indexed ona per-object basis. The object states may include object cache states.The object cache states may indicate locations of objects in any one orcombination of ARC, L2ARC, adaptive stage, local storage, and/or cloudstorage. By utilizing the mapping 406, the cloud interface device 402may directly identify the location of the one or more objects ofinterest. In some embodiments, the cloud interface device 402 may onlyutilize the mapping 406 in the event that there is no hit pursuant tothe ARC checking.

In some embodiments, in addition or in alternative to the caching, theintelligent pool management include keeping a mirror that continuouslysyncs with the cloud object storage 404. At least partially bysupporting cloud object data stores as virtual devices, certainembodiments may provide mirroring between local and cloud storage.Mirroring cloud storage with local storage may enable local storage readperformance while retaining replicated copies of all data in the cloud.Through the use of the mirror, the benefit of full performance may beprovided to the local system without having to keep multiple copieslocally. If any local data is missing or corrupt, the cloud copy of thedata may be accessed. The synchronous mirroring cloud and local devicesmay facilitate higher levels of performance.

To facilitate such synchronous mirroring, certain embodiments mayinclude a mirror VDEV. FIG. 14 illustrates an instance of an examplenetwork file system 200-2 of the hybrid cloud storage system 400 tofacilitate synchronous mirroring, in accordance with certain embodimentsof the present disclosure. The file system 200-2 may correspond to thefile system 200-1, but with mirror management integrated directly into aZFS control stack. Beyond that which is disclosed with respect to thefile system 200-1, the file system 200-2 may include a mirror VDEV 1402that facilitates cloud copies of data but local access times/speeds toread the data.

In some embodiments, the mirror VDEV 1402 may correspond to one or moreVDEVs of another VDEV type of a device driver interface inside a ZFSfile system architecture. The ZFS may communicate directly with themirror VDEV 1402, which may be at a virtual device layer directly abovea driver layer of the file system 200-2 and which, in some embodiments,correspond to an abstraction of the device driver interface inside theZFS architecture. The file system 200-2 may create the mirror VDEV 1402to be a funnel for I/O operations. In some embodiments, the mirror VDEV1402 may be a point with which other components of the file system 200-2may communicate primarily. For example, in some embodiments,communications from the transactional object layer may go through themirror VDEV 1402 to the physical layer. More specifically,communications from the DMU 218 may be directed to the I/O pipeline 224and to the mirror VDEV 1402. Responsive to such communications, themirror VDEV 1402 may direct communications to other VDEVs, such as theVDEV 226 and the cloud interface device 502. As such, the mirror VDEV1402 may coordinate I/O operations with respect to local storage 228 andcloud object storage 404.

In some embodiments, the mirror VDEV 1402 may only coordinate writeoperations with respect to local storage 228 and cloud object storage404 such that read operations need not go through the mirror VDEV 1402.With such embodiments, the other VDEVs, such as the VDEV 226 and thecloud interface device 502, may bypass the mirror VDEV 1402 for readoperations. In alternative embodiments, the mirror VDEV 1402 maycoordinate all I/O operations.

Advantageously, the mirror VDEV 1402 may coordinate write operations sothat each write operation is synchronously performed with respect tolocal storage 228 via one or more VDEVs 226 and with respect to thecloud object store 404 via the cloud interface device 502. Thissynchronous mirroring of each I/O operation is performed at the objectlevel, not the file level. The data replications with each I/O operationenables the hybrid cloud storage system 400 to achieve local storageread performance that masks latencies for cloud access. As a default,the hybrid cloud storage system 400 may read from local storage 228 inorder to avoid paying cloud latencies for the vast majority of readoperations. Only when the hybrid cloud storage system 400 determinesthat local data is missing or corrupt does the hybrid cloud storagesystem 400 need to access the cloud object store 404 to read a cloudcopy of the desired data. Such exceptions may be performed on an objectbasis so that the latency for cloud access is minimized.

FIG. 15 is a block diagram that illustrates an example method 1500directed to certain features of the hybrid cloud storage system 400 forsynchronous mirroring and cloud latency masking, in accordance withcertain embodiments of the present disclosure. According to certainembodiments, the method 1500 may begin as indicated by block 1502.However, as clarified above, certain steps of methods disclosed hereinmay be shuffled, combined, and/or performed simultaneously orsubstantially simultaneously in any suitable manner and may depend onthe implementation chosen.

As indicated by block 1502, POSIX-compliant request(s) to performparticular operation(s) (i.e., a transaction(s)) may be received fromthe application 202. Such an operation may correspond to writing and/ormodifying data. As indicated by block 1504, the POSIX-compliantrequest(s) may be forwarded from the operating system, via the systemcall interface 208, to the DMU 218. As indicated by block 1506, the DMU218 may translate requests to perform operations on data objectsdirectly to requests to perform write operations (i.e., I/O requests)directed to a physical location within the system storage pool 416 andthe cloud object store 404. The DMU 218 may forward the I/O requests tothe SPA.

The mirror VDEV 1402 of the SPA may receive the I/O requests (e.g., byway of the ARC 222 and the I/O pipeline 224). As indicated by block1508, the mirror VDEV 1402 may initiate writing of a data object withsynchronous replication on a per-I/O operation basis. One part of themirror VDEV 1402 may point to local storage, and one part of the mirrorVDEV 1402 may point to the cloud interface device 502 of the cloudstorage appliance 402. As indicated by block 1510, the mirror VDEV 1402may direct a first instance of the write operation to one or more of theVDEVs 226. In some embodiments, as indicated by block 1514, the COWprocedure disclosed above (e.g., in view of FIGS. 3A-3D) may proceed inorder to write the data object to the local system storage.

As indicated by block 1512, the mirror VDEV 1402 may direct a secondinstance of the write operation to the cloud interface device 502 of thecloud storage appliance 402. In some embodiments, as indicated by block1516, the COW procedure disclosed above may proceed in order to writethe data object to the local system storage. For example, the method1500 may transition to block 712 or another step of the method 700.

With each write operation synchronously performed with respect to localstorage 228 and with respect to the cloud object store 404, the hybridcloud storage system 400 may thereafter intelligently coordinate readoperations in order to achieve local storage read performance that maskslatencies for cloud access. At any suitable time after consummation ofthe replicative data storage, the hybrid cloud storage system 400 maycoordinate such read operations. Referring again to FIG. 15, asindicated by block 1518, a POSIX-compliant request to perform one ormore particular operations (i.e., one or more transactions) may bereceived from the application 202. Such an operation may correspond toreading or otherwise accessing data. As indicated by block 1520, thePOSIX-compliant request may be forwarded from the operating system, viathe system call interface 208, to the DMU 218. As indicated by block1522, the DMU 218 may translate requests to perform operations on dataobjects directly to requests to perform one or more read operations(i.e., one or more I/O requests) directed to the local storage 228. TheSPA may receive the I/O request(s) from the DMU 218. As indicated byblock 1524, responsive to the request(s), the SPA may initiate readingof one or more data objects from the local storage 228. In someembodiments, the ARC 222 may be checked first for a cached version ofthe one or more data objects and, absent a cached version being found,an attempt to read the one or more data objects from the local storage228 may then be made.

As indicated by block 1526, it may be determined whether one or morevalidated data objects corresponding to the I/O request(s) exist. Thismay include the SPA first determining if one or more objectscorresponding to the one or more I/O request(s) are retrievable from thelocal storage 228. Then, if such one or more objects are retrievable,the object(s) may be checked with a checksum(s) from parent node(s) inthe logical tree. In various embodiments, the checking may be performedby one or more of the VDEVs 226, the mirror VDEV 1402, and/or the I/Opipeline 224. As indicated by block 1528, in the case of the dataobject(s) being validated, the reading and/or further processingoperations of the system 400 may proceed, as the data has beendetermined to be not corrupted and not the incorrect version.

However, in the case of the SPA determining that validated data objectscorresponding to the I/O request(s) does not exist, process flow maytransition to block 1530. Such a determination could correspond to anabsence of retrievable data objects corresponding to the one or more I/Orequest(s), in which case an error condition may be identified.Similarly, such a determination could correspond to mismatch of actualand expected checksums of the data object(s) corresponding to the one ormore I/O request(s), in which case an error condition may also beidentified. In either case, the SPA may initiate reading of the one ormore data objects from the cloud object store 404, as indicated by block1530. In various embodiments, the initiating of the reading may beperformed by one or a combination of the DMU 218, the mirror VDEV 1402,the I/O pipeline 224, and/or the cloud interface appliance 402.

The reading of the one or more data objects from the cloud object store404 may include steps previously disclosed herein, for example, withregard to FIG. 9. Such steps may include one or a combination of issuingI/O request(s) to the cloud interface appliance 402, sendingcorresponding cloud interface request(s) to the cloud object store 404using the mapping 406 of cloud storage objects 414, receiving dataobject(s) responsive to the object interface requests, and the like,which are detailed above. As indicated by block 1532, it may bedetermined whether a validated data object has been retrieved from thecloud object store 404. Again, this may involve steps previouslydisclosed herein, for example, with regard to FIG. 9, where it isdetermined whether to validate the data object(s) by the checksum(s)from parent node(s) in the logical tree.

In the case of the data being validated, the process flow may transitionto block 1528, and the reading and/or further processing operations ofthe system 400 may proceed. Further, as indicated by block 1534, thecorrect data may be written to the local system storage. In variousembodiments, the correction process may be performed by one or acombination of the DMU 218, the mirror VDEV 1402, the I/O pipeline 224,and/or the cloud interface appliance 402. In some embodiments, theprocess flow may transition to block 1514, where a COW procedure mayproceed in order to write the correct data to the local system storage.

However, in the case of mismatch of actual and expected checksums of thedata object(s), the process flow may transition to block 1536, whereremediation processes may be initiated. This may involve stepspreviously disclosed herein, for example, with regard to FIG. 9, whereremediation processes are disclosed. For example, the remediationprocesses may include reissuing one or more cloud interface requests,requesting the correct version of the data from another cloud objectstore, and/or the like.

Advantageously, when an amount of data maintained by the hybrid cloudstorage system 400 exceeds a certain amount, it may become morecost-benefit optimal to transition from a mirroring mode disclosed aboveto a caching mode in accordance with embodiments disclosed with respectto FIGS. 12 and 13. Certain embodiments may make that transitionautomatically. The hybrid cloud storage system 400 may begin withmirroring techniques, proceeding until one or more thresholds arereached. Such thresholds could be defined in terms of storage usage. Forexample, when the usage of local storage capacity reaches a thresholdpercentage (or absolute value, relative value, etc.), the hybrid cloudstorage system 400 may transition to adaptive I/O staging. In that way,the hybrid cloud storage system 400 may balance the load imparted on thelocal storage by shifting operational modes. Then, the hybrid cloudstorage system 400 may the most relevant X amount of data (e.g., 10 TB,and/or the like) and shunt the rest of the data to the cloud storage.This load balancing may allow for fewer storage devices, whileaccommodating increasing amounts of data storage.

FIG. 16 depicts a simplified diagram of a distributed system 1600 forimplementing certain embodiments in accordance with present disclosure.In the illustrated embodiment, distributed system 1600 includes one ormore client computing devices 1602, 1604, 1606, and 1608, which areconfigured to execute and operate a client application such as a webbrowser, proprietary client (e.g., Oracle Forms), or the like over oneor more network(s) 1610. Server 1612 may be communicatively coupled withremote client computing devices 1602, 1604, 1606, and 1608 via network1610.

In various embodiments, server 1612 may be adapted to run one or moreservices or software applications provided by one or more of thecomponents of the system. In some embodiments, these services may beoffered as web-based or cloud services or under a Software as a Service(SaaS) model to the users of client computing devices 1602, 1604, 1606,and/or 1608. Users operating client computing devices 1602, 1604, 1606,and/or 1608 may in turn utilize one or more client applications tointeract with server 1612 to utilize the services provided by thesecomponents.

In the configuration depicted in the figure, the software components1618, 1620 and 1622 of system 1600 are shown as being implemented onserver 1612. In other embodiments, one or more of the components ofsystem 1600 and/or the services provided by these components may also beimplemented by one or more of the client computing devices 1602, 1604,1606, and/or 1608. Users operating the client computing devices may thenutilize one or more client applications to use the services provided bythese components. These components may be implemented in hardware,firmware, software, or combinations thereof. It should be appreciatedthat various different system configurations are possible, which may bedifferent from distributed system 1600. The embodiment shown in thefigure is thus one example of a distributed system for implementing anembodiment system and is not intended to be limiting.

Client computing devices 1602, 1604, 1606, and/or 1608 may be portablehandheld devices (e.g., an iPhone®, cellular telephone, an iPad®,computing tablet, a personal digital assistant (PDA)) or wearabledevices (e.g., a Google Glass® head-mounted display), running softwaresuch as Microsoft Windows Mobile®, and/or a variety of mobile operatingsystems such as iOS, Windows Phone, Android, BlackBerry, Palm OS, andthe like, and being Internet, e-mail, short message service (SMS),Blackberry®, or other communication protocol enabled. The clientcomputing devices can be general purpose personal computers including,by way of example, personal computers and/or laptop computers runningvarious versions of Microsoft Windows®, Apple Macintosh®, and/or Linuxoperating systems. The client computing devices can be workstationcomputers running any of a variety of commercially-available UNIX® orUNIX-like operating systems, including without limitation the variety ofGNU/Linux operating systems, such as for example, Google Chrome OS.Alternatively, or in addition, client computing devices 1602, 1604,1606, and 1608 may be any other electronic device, such as a thin-clientcomputer, an Internet-enabled gaming system (e.g., a Microsoft Xboxgaming console with or without a Kinect® gesture input device), and/or apersonal messaging device, capable of communicating over network(s)1610.

Although exemplary distributed system 1600 is shown with four clientcomputing devices, any number of client computing devices may besupported. Other devices, such as devices with sensors, etc., mayinteract with server 1612.

Network(s) 1610 in distributed system 1600 may be any type of networkfamiliar to those skilled in the art that can support datacommunications using any of a variety of commercially-availableprotocols, including without limitation TCP/IP (transmission controlprotocol/Internet protocol), SNA (systems network architecture), IPX(Internet packet exchange), AppleTalk, and the like. Merely by way ofexample, network(s) 1610 can be a local area network (LAN), such as onebased on Ethernet, Token-Ring and/or the like. Network(s) 1610 can be awide-area network and the Internet. It can include a virtual network,including without limitation a virtual private network (VPN), anintranet, an extranet, a public switched telephone network (PSTN), aninfra-red network, a wireless network (e.g., a network operating underany of the Institute of Electrical and Electronics (IEEE) 802.11 suiteof protocols, Bluetooth®, and/or any other wireless protocol); and/orany combination of these and/or other networks.

Server 1612 may be composed of one or more general purpose computers,specialized server computers (including, by way of example, PC (personalcomputer) servers, UNIX® servers, mid-range servers, mainframecomputers, rack-mounted servers, etc.), server farms, server clusters,or any other appropriate arrangement and/or combination. In variousembodiments, server 1612 may be adapted to run one or more services orsoftware applications described in the foregoing disclosure. Forexample, server 1612 may correspond to a server for performingprocessing described above according to an embodiment of the presentdisclosure.

Server 1612 may run an operating system including any of those discussedabove, as well as any commercially available server operating system.Server 1612 may also run any of a variety of additional serverapplications and/or mid-tier applications, including HTTP (hypertexttransport protocol) servers, FTP (file transfer protocol) servers, CGI(common gateway interface) servers, JAVA® servers, database servers, andthe like. Exemplary database servers include without limitation thosecommercially available from Oracle, Microsoft, Sybase, IBM(International Business Machines), and the like.

In some implementations, server 1612 may include one or moreapplications to analyze and consolidate data feeds and/or event updatesreceived from users of client computing devices 1602, 1604, 1606, and1608. As an example, data feeds and/or event updates may include, butare not limited to, Twitter® feeds, Facebook® updates or real-timeupdates received from one or more third party information sources andcontinuous data streams, which may include real-time events related tosensor data applications, financial tickers, network performancemeasuring tools (e.g., network monitoring and traffic managementapplications), clickstream analysis tools, automobile trafficmonitoring, and the like. Server 1612 may also include one or moreapplications to display the data feeds and/or real-time events via oneor more display devices of client computing devices 1602, 1604, 1606,and 1608.

Distributed system 1600 may also include one or more databases 1614 and1616. Databases 1614 and 1616 may reside in a variety of locations. Byway of example, one or more of databases 1614 and 1616 may reside on anon-transitory storage medium local to (and/or resident in) server 1612.Alternatively, databases 1614 and 1616 may be remote from server 1612and in communication with server 1612 via a network-based or dedicatedconnection. In one set of embodiments, databases 1614 and 1616 mayreside in a storage-area network (SAN). Similarly, any necessary filesfor performing the functions attributed to server 1612 may be storedlocally on server 1612 and/or remotely, as appropriate. In one set ofembodiments, databases 1614 and 1616 may include relational databases,such as databases provided by Oracle, that are adapted to store, update,and retrieve data in response to SQL-formatted commands.

FIG. 17 is a simplified block diagram of one or more components of asystem environment 1700 by which services provided by one or morecomponents of a system may be offered as cloud services, in accordancewith certain embodiments of the present disclosure. In the illustratedembodiment, system environment 1700 includes one or more clientcomputing devices 1704, 1706, and 1708 that may be used by users tointeract with a cloud infrastructure system 1702 that provides cloudservices. The client computing devices may be configured to operate aclient application such as a web browser, a proprietary clientapplication (e.g., Oracle Forms), or some other application, which maybe used by a user of the client computing device to interact with cloudinfrastructure system 1702 to use services provided by cloudinfrastructure system 1702.

It should be appreciated that cloud infrastructure system 1702 depictedin the figure may have other components than those depicted. Further,the embodiment shown in the figure is only one example of a cloudinfrastructure system that may incorporate an embodiment of theinvention. In some other embodiments, cloud infrastructure system 1702may have more or fewer components than shown in the figure, may combinetwo or more components, or may have a different configuration orarrangement of components.

Client computing devices 1704, 1706, and 1708 may be devices similar tothose described above for 1602, 1604, 1606, and 1608. Although exemplarysystem environment 1700 is shown with three client computing devices,any number of client computing devices may be supported. Other devicessuch as devices with sensors, etc. may interact with cloudinfrastructure system 1702.

Network(s) 1710 may facilitate communications and exchange of databetween clients 1704, 1706, and 1708 and cloud infrastructure system1702. Each network may be any type of network familiar to those skilledin the art that can support data communications using any of a varietyof commercially-available protocols, including those described above fornetwork(s) 1610. Cloud infrastructure system 1702 may comprise one ormore computers and/or servers that may include those described above forserver 1612.

In certain embodiments, services provided by the cloud infrastructuresystem may include a host of services that are made available to usersof the cloud infrastructure system on demand, such as online datastorage and backup solutions, Web-based e-mail services, hosted officesuites and document collaboration services, database processing, managedtechnical support services, and the like. Services provided by the cloudinfrastructure system can dynamically scale to meet the needs of itsusers. A specific instantiation of a service provided by cloudinfrastructure system is referred to herein as a “service instance.” Ingeneral, any service made available to a user via a communicationnetwork, such as the Internet, from a cloud service provider's system isreferred to as a “cloud service.” Typically, in a public cloudenvironment, servers and systems that make up the cloud serviceprovider's system are different from the customer's own on-premisesservers and systems. For example, a cloud service provider's system mayhost an application, and a user may, via a communication network such asthe Internet, on demand, order and use the application.

In some examples, a service in a computer network cloud infrastructuremay include protected computer network access to storage, a hosteddatabase, a hosted web server, a software application, or other serviceprovided by a cloud vendor to a user, or as otherwise known in the art.For example, a service can include password-protected access to remotestorage on the cloud through the Internet. As another example, a servicecan include a web service-based hosted relational database and ascript-language middleware engine for private use by a networkeddeveloper. As another example, a service can include access to an emailsoftware application hosted on a cloud vendor's web site.

In certain embodiments, cloud infrastructure system 1702 may include asuite of applications, middleware, and database service offerings thatare delivered to a customer in a self-service, subscription-based,elastically scalable, reliable, highly available, and secure manner. Anexample of such a cloud infrastructure system is the Oracle Public Cloudprovided by the present assignee.

In various embodiments, cloud infrastructure system 1702 may be adaptedto automatically provision, manage and track a customer's subscriptionto services offered by cloud infrastructure system 1702. Cloudinfrastructure system 1702 may provide the cloud services via differentdeployment models. For example, services may be provided under a publiccloud model in which cloud infrastructure system 1702 is owned by anorganization selling cloud services (e.g., owned by Oracle) and theservices are made available to the general public or different industryenterprises. As another example, services may be provided under aprivate cloud model in which cloud infrastructure system 1702 isoperated solely for a single organization and may provide services forone or more entities within the organization. The cloud services mayalso be provided under a community cloud model in which cloudinfrastructure system 1702 and the services provided by cloudinfrastructure system 1702 are shared by several organizations in arelated community. The cloud services may also be provided under ahybrid cloud model, which is a combination of two or more differentmodels.

In some embodiments, the services provided by cloud infrastructuresystem 1702 may include one or more services provided under Software asa Service (SaaS) category, Platform as a Service (PaaS) category,Infrastructure as a Service (IaaS) category, or other categories ofservices including hybrid services. A customer, via a subscriptionorder, may order one or more services provided by cloud infrastructuresystem 1702. Cloud infrastructure system 1702 then performs processingto provide the services in the customer's subscription order.

In some embodiments, the services provided by cloud infrastructuresystem 1702 may include, without limitation, application services,platform services and infrastructure services. In some examples,application services may be provided by the cloud infrastructure systemvia a SaaS platform. The SaaS platform may be configured to providecloud services that fall under the SaaS category. For example, the SaaSplatform may provide capabilities to build and deliver a suite ofon-demand applications on an integrated development and deploymentplatform. The SaaS platform may manage and control the underlyingsoftware and infrastructure for providing the SaaS services. Byutilizing the services provided by the SaaS platform, customers canutilize applications executing on the cloud infrastructure system.Customers can acquire the application services without the need forcustomers to purchase separate licenses and support. Various differentSaaS services may be provided. Examples include, without limitation,services that provide solutions for sales performance management,enterprise integration, and business flexibility for largeorganizations.

In some embodiments, platform services may be provided by the cloudinfrastructure system via a PaaS platform. The PaaS platform may beconfigured to provide cloud services that fall under the PaaS category.Examples of platform services may include without limitation servicesthat enable organizations (such as Oracle) to consolidate existingapplications on a shared, common architecture, as well as the ability tobuild new applications that leverage the shared services provided by theplatform. The PaaS platform may manage and control the underlyingsoftware and infrastructure for providing the PaaS services. Customerscan acquire the PaaS services provided by the cloud infrastructuresystem without the need for customers to purchase separate licenses andsupport. Examples of platform services include, without limitation,Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS),and others.

By utilizing the services provided by the PaaS platform, customers canemploy programming languages and tools supported by the cloudinfrastructure system and also control the deployed services. In someembodiments, platform services provided by the cloud infrastructuresystem may include database cloud services, middleware cloud services(e.g., Oracle Fusion Middleware services), and Java cloud services. Inone embodiment, database cloud services may support shared servicedeployment models that enable organizations to pool database resourcesand offer customers a Database as a Service in the form of a databasecloud. Middleware cloud services may provide a platform for customers todevelop and deploy various business applications, and Java cloudservices may provide a platform for customers to deploy Javaapplications, in the cloud infrastructure system.

Various different infrastructure services may be provided by an IaaSplatform in the cloud infrastructure system. The infrastructure servicesfacilitate the management and control of the underlying computingresources, such as storage, networks, and other fundamental computingresources for customers utilizing services provided by the SaaS platformand the PaaS platform.

In certain embodiments, cloud infrastructure system 1702 may alsoinclude infrastructure resources 1730 for providing the resources usedto provide various services to customers of the cloud infrastructuresystem. In one embodiment, infrastructure resources 1730 may includepre-integrated and optimized combinations of hardware, such as servers,storage, and networking resources to execute the services provided bythe PaaS platform and the SaaS platform. In some embodiments, resourcesin cloud infrastructure system 1702 may be shared by multiple users anddynamically re-allocated per demand. Additionally, resources may beallocated to users in different time zones. For example, cloudinfrastructure system 1730 may enable a first set of users in a firsttime zone to utilize resources of the cloud infrastructure system for aspecified number of hours and then enable the re-allocation of the sameresources to another set of users located in a different time zone,thereby maximizing the utilization of resources.

In certain embodiments, a number of internal shared services 1732 may beprovided that are shared by different components or modules of cloudinfrastructure system 1702 and by the services provided by cloudinfrastructure system 1702. These internal shared services may include,without limitation, a security and identity service, an integrationservice, an enterprise repository service, an enterprise managerservice, a virus scanning and white list service, a high availability,backup and recovery service, service for enabling cloud support, anemail service, a notification service, a file transfer service, and thelike. In certain embodiments, cloud infrastructure system 1702 mayprovide comprehensive management of cloud services (e.g., SaaS, PaaS,and IaaS services) in the cloud infrastructure system. In oneembodiment, cloud management functionality may include capabilities forprovisioning, managing and tracking a customer's subscription receivedby cloud infrastructure system 1702, and the like.

In certain embodiments, as depicted in the figure, cloud managementfunctionality may be provided by one or more modules, such as an ordermanagement module 1720, an order orchestration module 1722, an orderprovisioning module 1724, an order management and monitoring module1726, and an identity management module 1728. These modules may includeor be provided using one or more computers and/or servers, which may begeneral purpose computers, specialized server computers, server farms,server clusters, or any other appropriate arrangement and/orcombination.

In exemplary operation 1734, a customer using a client device, such asclient device 1704, 1706 or 1708, may interact with cloud infrastructuresystem 1702 by requesting one or more services provided by cloudinfrastructure system 1702 and placing an order for a subscription forone or more services offered by cloud infrastructure system 1702. Incertain embodiments, the customer may access a cloud User Interface(UI), cloud UI 1712, cloud UI 1714 and/or cloud UI 1716 and place asubscription order via these UIs. The order information received bycloud infrastructure system 1702 in response to the customer placing anorder may include information identifying the customer and one or moreservices offered by the cloud infrastructure system 1702 that thecustomer intends to subscribe to.

After an order has been placed by the customer, the order information isreceived via the cloud UIs, 1712, 1714 and/or 1716. At operation 1736,the order is stored in order database 1718. Order database 1718 can beone of several databases operated by cloud infrastructure system 1718and operated in conjunction with other system elements. At operation1738, the order information is forwarded to an order management module1720. In some instances, order management module 1720 may be configuredto perform billing and accounting functions related to the order, suchas verifying the order, and upon verification, booking the order.

At operation 1740, information regarding the order is communicated to anorder orchestration module 1722. Order orchestration module 1722 mayutilize the order information to orchestrate the provisioning ofservices and resources for the order placed by the customer. In someinstances, order orchestration module 1722 may orchestrate theprovisioning of resources to support the subscribed services using theservices of order provisioning module 1724.

In certain embodiments, order orchestration module 1722 enables themanagement of business processes associated with each order and appliesbusiness logic to determine whether an order should proceed toprovisioning. At operation 1742, upon receiving an order for a newsubscription, order orchestration module 1722 sends a request to orderprovisioning module 1724 to allocate resources and configure thoseresources needed to fulfill the subscription order. Order provisioningmodule 1724 enables the allocation of resources for the services orderedby the customer. Order provisioning module 1724 provides a level ofabstraction between the cloud services provided by cloud infrastructuresystem 1700 and the physical implementation layer that is used toprovision the resources for providing the requested services. Orderorchestration module 1722 may thus be isolated from implementationdetails, such as whether or not services and resources are actuallyprovisioned on the fly or pre-provisioned and only allocated/assignedupon request.

At operation 1744, once the services and resources are provisioned, anotification of the provided service may be sent to customers on clientdevices 1704, 1706, and/or 1708 by order provisioning module 1724 ofcloud infrastructure system 1702. At operation 1746, the customer'ssubscription order may be managed and tracked by an order management andmonitoring module 1726. In some instances, order management andmonitoring module 1726 may be configured to collect usage statistics forthe services in the subscription order, such as the amount of storageused, the amount data transferred, the number of users, and the amountof system up time and system down time.

In certain embodiments, cloud infrastructure system 1700 may include anidentity management module 1728. Identity management module 1728 may beconfigured to provide identity services, such as access management andauthorization services in cloud infrastructure system 1700. In someembodiments, identity management module 1728 may control informationabout customers who wish to utilize the services provided by cloudinfrastructure system 1702. Such information can include informationthat authenticates the identities of such customers and information thatdescribes which actions those customers are authorized to performrelative to various system resources (e.g., files, directories,applications, communication ports, memory segments, etc.). Identitymanagement module 1728 may also include the management of descriptiveinformation about each customer and about how and by whom thatdescriptive information can be accessed and modified.

FIG. 18 illustrates an exemplary computer system 1800, in which variousembodiments of the present invention may be implemented. The system 1800may be used to implement any of the computer systems described herein.As shown in the figure, computer system 1800 includes a processing unit1804 that communicates with a number of peripheral subsystems via a bussubsystem 1802. These peripheral subsystems may include a processingacceleration unit 1806, an I/O subsystem 1808, a storage subsystem 1818and a communications subsystem 1824. Storage subsystem 1818 includestangible computer-readable storage media 1822 and a system memory 1810.

Bus subsystem 1802 provides a mechanism for letting the variouscomponents and subsystems of computer system 1800 communicate with eachother as intended. Although bus subsystem 1802 is shown schematically asa single bus, alternative embodiments of the bus subsystem may utilizemultiple buses. Bus subsystem 1802 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Forexample, such architectures may include an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 1804, which can be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 1800. One or more processorsmay be included in processing unit 1804. These processors may includesingle core or multicore processors. In certain embodiments, processingunit 1804 may be implemented as one or more independent processing units1832 and/or 1834 with single or multicore processors included in eachprocessing unit. In other embodiments, processing unit 1804 may also beimplemented as a quad-core processing unit formed by integrating twodual-core processors into a single chip.

In various embodiments, processing unit 1804 can execute a variety ofprograms in response to program code and can maintain multipleconcurrently executing programs or processes. At any given time, some orall of the program code to be executed can be resident in processor(s)1804 and/or in storage subsystem 1818. Through suitable programming,processor(s) 1804 can provide various functionalities described above.Computer system 1800 may additionally include a processing accelerationunit 1806, which can include a digital signal processor (DSP), aspecial-purpose processor, and/or the like. In some embodiments, theprocessing acceleration unit 1806 may include or work in conjunctionwith an acceleration engine such as that disclosed herein to improvecomputer system functioning.

I/O subsystem 1808 may include user interface input devices and userinterface output devices. User interface input devices may include akeyboard, pointing devices such as a mouse or trackball, a touchpad ortouch screen incorporated into a display, a scroll wheel, a click wheel,a dial, a button, a switch, a keypad, audio input devices with voicecommand recognition systems, microphones, and other types of inputdevices. User interface input devices may include, for example, motionsensing and/or gesture recognition devices such as the Microsoft Kinect®motion sensor that enables users to control and interact with an inputdevice, such as the Microsoft Xbox® 360 game controller, through anatural user interface using gestures and spoken commands. Userinterface input devices may also include eye gesture recognition devicessuch as the Google Glass® blink detector that detects eye activity(e.g., ‘blinking’ while taking pictures and/or making a menu selection)from users and transforms the eye gestures as input into an input device(e.g., Google Glass®). Additionally, user interface input devices mayinclude voice recognition sensing devices that enable users to interactwith voice recognition systems (e.g., Siri® navigator), through voicecommands.

User interface input devices may also include, without limitation, threedimensional (3D) mice, joysticks or pointing sticks, gamepads andgraphic tablets, and audio/visual devices such as speakers, digitalcameras, digital camcorders, portable media players, webcams, imagescanners, fingerprint scanners, barcode reader 3D scanners, 3D printers,laser rangefinders, and eye gaze tracking devices. Additionally, userinterface input devices may include, for example, medical imaging inputdevices such as computed tomography, magnetic resonance imaging,position emission tomography, medical ultrasonography devices. Userinterface input devices may also include, for example, audio inputdevices such as MIDI keyboards, digital musical instruments and thelike.

User interface output devices may include a display subsystem, indicatorlights, or non-visual displays such as audio output devices, etc. Thedisplay subsystem may be a cathode ray tube (CRT), a flat-panel device,such as that using a liquid crystal display (LCD) or plasma display, aprojection device, a touch screen, and the like. In general, use of theterm “output device” is intended to include all possible types ofdevices and mechanisms for outputting information from computer system1800 to a user or other computer. For example, user interface outputdevices may include, without limitation, a variety of display devicesthat visually convey text, graphics and audio/video information such asmonitors, printers, speakers, headphones, automotive navigation systems,plotters, voice output devices, and modems.

Computer system 1800 may comprise a storage subsystem 1818 thatcomprises software elements, shown as being currently located within asystem memory 1810. System memory 1810 may store program instructionsthat are loadable and executable on processing unit 1804, as well asdata generated during the execution of these programs. Depending on theconfiguration and type of computer system 1800, system memory 1810 maybe volatile (such as random access memory (RAM)) and/or non-volatile(such as read-only memory (ROM), flash memory, etc.) The RAM typicallycontains data and/or program modules that are immediately accessible toand/or presently being operated and executed by processing unit 1804. Insome implementations, system memory 1810 may include multiple differenttypes of memory, such as static random access memory (SRAM) or dynamicrandom access memory (DRAM). In some implementations, a basicinput/output system (BIOS), containing the basic routines that help totransfer information between elements within computer system 1800, suchas during start-up, may typically be stored in the ROM. By way ofexample, and not limitation, system memory 1810 also illustratesapplication programs 1812, which may include client applications, Webbrowsers, mid-tier applications, relational database management systems(RDBMS), etc., program data 1814, and an operating system 1816. By wayof example, operating system 1816 may include various versions ofMicrosoft Windows®, Apple Macintosh®, and/or Linux operating systems, avariety of commercially-available UNIX® or UNIX-like operating systems(including without limitation the variety of GNU/Linux operatingsystems, the Google Chrome® OS, and the like) and/or mobile operatingsystems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, andPalm® OS operating systems.

Storage subsystem 1818 may also provide a tangible computer-readablestorage medium for storing the basic programming and data constructsthat provide the functionality of some embodiments. Software (programs,code modules, instructions) that when executed by a processor providethe functionality described above may be stored in storage subsystem1818. These software modules or instructions may be executed byprocessing unit 1804. Storage subsystem 1818 may also provide arepository for storing data used in accordance with the presentinvention.

Storage subsystem 1800 may also include a computer-readable storagemedia reader 1820 that can further be connected to computer-readablestorage media 1822. Together and, optionally, in combination with systemmemory 1810, computer-readable storage media 1822 may comprehensivelyrepresent remote, local, fixed, and/or removable storage devices plusstorage media for temporarily and/or more permanently containing,storing, transmitting, and retrieving computer-readable information.

Computer-readable storage media 1822 containing code, or portions ofcode, can also include any appropriate media known or used in the art,including storage media and communication media, such as but not limitedto, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible computer-readable storagemedia such as RAM, ROM, electronically erasable programmable ROM(EEPROM), flash memory or other memory technology, CD-ROM, digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or other tangible computer readable media. This can also includenontangible computer-readable media, such as data signals, datatransmissions, or any other medium which can be used to transmit thedesired information and which can be accessed by computing system 1800.

By way of example, computer-readable storage media 1822 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 1822 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 1822 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magneto resistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 1800.

Communications subsystem 1824 provides an interface to other computersystems and networks. Communications subsystem 1824 serves as aninterface for receiving data from and transmitting data to other systemsfrom computer system 1800. For example, communications subsystem 1824may enable computer system 1800 to connect to one or more devices viathe Internet. In some embodiments communications subsystem 1824 caninclude radio frequency (RF) transceiver components for accessingwireless voice and/or data networks (e.g., using cellular telephonetechnology, advanced data network technology, such as 18G, 4G or EDGE(enhanced data rates for global evolution), WiFi (IEEE 802.11 familystandards, or other mobile communication technologies, or anycombination thereof), global positioning system (GPS) receivercomponents, and/or other components. In some embodiments communicationssubsystem 1824 can provide wired network connectivity (e.g., Ethernet)in addition to or instead of a wireless interface.

In some embodiments, communications subsystem 1824 may also receiveinput communication in the form of structured and/or unstructured datafeeds 1826, event streams 1828, event updates 1830, and the like onbehalf of one or more users who may use computer system 1800. By way ofexample, communications subsystem 1824 may be configured to receive datafeeds 1826 in real-time from users of social networks and/or othercommunication services such as Twitter® feeds, Facebook® updates, webfeeds such as Rich Site Summary (RSS) feeds, and/or real-time updatesfrom one or more third party information sources.

Additionally, communications subsystem 1824 may also be configured toreceive data in the form of continuous data streams, which may includeevent streams 1828 of real-time events and/or event updates 1830, thatmay be continuous or unbounded in nature with no explicit end. Examplesof applications that generate continuous data may include, for example,sensor data applications, financial tickers, network performancemeasuring tools (e.g., network monitoring and traffic managementapplications), clickstream analysis tools, automobile trafficmonitoring, and the like. Communications subsystem 1824 may also beconfigured to output the structured and/or unstructured data feeds 1826,event streams 1828, event updates 1830, and the like to one or moredatabases that may be in communication with one or more streaming datasource computers coupled to computer system 1800.

Computer system 1800 can be one of various types, including a handheldportable device (e.g., an iPhone® cellular phone, an iPad® computingtablet, a PDA), a wearable device (e.g., a Google Glass® head mounteddisplay), a PC, a workstation, a mainframe, a kiosk, a server rack, orany other data processing system. Due to the ever-changing nature ofcomputers and networks, the description of computer system 1800 depictedin the figure is intended only as a specific example. Many otherconfigurations having more or fewer components than the system depictedin the figure are possible. For example, customized hardware might alsobe used and/or particular elements might be implemented in hardware,firmware, software (including applets), or a combination. Further,connection to other computing devices, such as network input/outputdevices, may be employed. Based on the disclosure and teachings providedherein, a person of ordinary skill in the art will appreciate other waysand/or methods to implement the various embodiments.

In the foregoing description, for the purposes of explanation, numerousspecific details were set forth in order to provide a thoroughunderstanding of various embodiments of the present invention. It willbe apparent, however, to one skilled in the art that embodiments of thepresent invention may be practiced without some of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form.

The foregoing description provides exemplary embodiments only, and isnot intended to limit the scope, applicability, or configuration of thedisclosure. Rather, the foregoing description of the exemplaryembodiments will provide those skilled in the art with an enablingdescription for implementing an exemplary embodiment. It should beunderstood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope ofthe invention as set forth in the appended claims.

Specific details are given in the foregoing description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits,systems, networks, processes, and other components may have been shownas components in block diagram form in order not to obscure theembodiments in unnecessary detail. In other instances, well-knowncircuits, processes, algorithms, structures, and techniques may havebeen shown without unnecessary detail in order to avoid obscuring theembodiments.

Also, it is noted that individual embodiments may have been described asa process which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay have described the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations may be re-arranged. A process is terminatedwhen its operations are completed, but could have additional steps notincluded in a figure. A process may correspond to a method, a function,a procedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

The term “computer-readable medium” includes, but is not limited toportable or fixed storage devices, optical storage devices, wirelesschannels and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A code segment ormachine-executable instructions may represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc., may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks may be stored in a machine readable medium. A processor(s) mayperform the necessary tasks.

In the foregoing specification, aspects of the invention are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the invention is not limited thereto. Variousfeatures and aspects of the above-described invention may be usedindividually or jointly. Further, embodiments can be utilized in anynumber of environments and applications beyond those described hereinwithout departing from the broader spirit and scope of thespecification. The specification and drawings are, accordingly, to beregarded as illustrative rather than restrictive.

Additionally, for the purposes of illustration, methods were describedin a particular order. It should be appreciated that in alternateembodiments, the methods may be performed in a different order than thatdescribed. It should also be appreciated that the methods describedabove may be performed by hardware components or may be embodied insequences of machine-executable instructions, which may be used to causea machine, such as a general-purpose or special-purpose processor orlogic circuits programmed with the instructions to perform the methods.These machine-executable instructions may be stored on one or moremachine readable mediums, such as CD-ROMs or other type of opticaldisks, floppy diskettes, ROMs, RAMs, EPROMs, EEPROMs, magnetic oroptical cards, flash memory, or other types of machine-readable mediumssuitable for storing electronic instructions. Alternatively, the methodsmay be performed by a combination of hardware and software.

Also, the terms in the claims have their plain, ordinary meaning unlessotherwise explicitly and clearly defined by the patentee. The indefinitearticles “a” or “an,” as used in the claims, are defined herein to meanone or more than one of the element that the particular articleintroduces; and subsequent use of the definite article “the” is notintended to negate that meaning. Furthermore, the use of ordinal numberterms, such as “first,” “second,” etc., to clarify different elements inthe claims is not intended to impart a particular position in a series,or any other sequential character or order, to the elements to which theordinal number terms have been applied.

What is claimed:
 1. A method comprising: receiving, from an applicationlayer of a Z File System (ZFS) system and through a system callinterface of an interface layer of the ZFS system, a file; creating, viaone or more virtual devices of a plurality of virtual devices at avirtual device layer of the ZFS system, data blocks that correspond tothe file; causing, by a cloud interface appliance, storage of cloudstorage objects in a cloud object store, the cloud storage objectscomprising data of the data blocks and corresponding metadata ofaccording to a tree hierarchy, wherein: a first subset of the cloudstorage objects comprises the data of the data blocks, the first subsetcorresponding to leaf nodes of the tree hierarchy; a second subset ofthe cloud storage objects comprises the corresponding metadata, thesecond subset corresponding to non-leaf nodes of the tree hierarchy; andthe corresponding metadata comprises respective checksums generated foreach cloud storage object of the first subset of the cloud storageobjects; receiving, from the application layer of the ZFS system, arequest to perform a transaction with respect to the file; translating,by a data management unit of a transactional object layer of the ZFSsystem, the request into an I/O request; based at least in part on theI/O request, performing, by the cloud interface appliance, a writeoperation to modify the tree hierarchy at least in part by causingstorage of an additional cloud storage objects to result in a modifiedtree hierarchy, wherein: a third subset of the additional cloud storageobjects comprises additional data, the third subset corresponding to oneor more additional leaf nodes as modifications to the tree hierarchy; afourth subset of the additional cloud storage objects comprisesadditional metadata corresponding to the additional data, the fourthsubset corresponding to additional non-leaf nodes of the tree hierarchy;and the additional metadata comprises additional respective checksumsgenerated for each cloud storage object of the third subset of theadditional cloud storage objects; based at least in part on a subsequentI/O request, performing a read operation with respect to the modifiedtree hierarchy, wherein the read operation comprises receiving a versionof at least one cloud storage object from the cloud object store; andchecking the version of the at least one cloud storage object with areference checksum that corresponds to at least one of the respectivechecksums or at least one of the additional respective checksums, wherethe reference checksum is stored locally with respect to the ZFS systemand/or the cloud interface appliance or is received from the cloudobject store.
 2. The method of claim 1, further comprising: receiving,from the application layer of the ZFS system, a subsequent request toperform a subsequent transaction with respect to the file; andtranslating, by the data management unit, the subsequent request intothe subsequent I/O request.
 3. The method of claim 2, wherein thechecking the version of the at least one cloud storage object with thereference checksum comprises: processing, by a storage pool allocator orthe cloud interface appliance, the version of the at least one cloudstorage object to generate a generated checksum; retrieving, by thestorage pool allocator or the cloud interface appliance, specificmetadata that is stored separately from the at least one cloud storageobject and that corresponds to a specific non-leaf node of the non-leafnodes or additional non-leaf nodes according to the modified treehierarchy, wherein the specific non-leaf node is a parent node withrespect to the at least one cloud storage object; identifying, by thestorage pool allocator or the cloud interface appliance, from thespecific metadata the reference checksum for the at least one cloudstorage object, the reference checksum being associated with a lastmodified version of the cloud storage object; and determining whetherthe generated checksum matches the reference checksum.
 4. The method ofclaim 3, wherein the performing, by the cloud interface appliance, thewrite operation to modify the tree hierarchy is performed according to acopy-on-write process.
 5. The method of claim 4, further comprising:causing, via the cloud interface appliance or another cloud interfaceappliance, storage of the cloud storage objects and the additional cloudstorage objects in second cloud object store so that a second cloudinstance of the modified tree hierarchy is stored in the second cloudobject store, wherein the modified tree hierarchy stored in the cloudobject store corresponds to a first cloud instance of the modified treehierarchy; wherein the causing storage of the cloud storage objects andthe additional cloud storage objects in the second cloud object storecomprises independent generating additional checksums for each cloudstorage object stored as leaf nodes in the second cloud object store. 6.The method of claim 5, further comprising: after the storage of thecloud storage objects and the additional cloud storage objects in thesecond cloud object store, identifying an unsynchronized state of thefirst cloud instance of the tree hierarchy with respect to the secondcloud instance of the tree hierarchy based at least in part on at leastone checksum stored in at least one non-leaf node of the first cloudinstance and at least one checksum stored in at least one non-leaf nodeof the second cloud instance.
 7. The method of claim 6, furthercomprising: in response to identifying the unsynchronized state,initiating, by the ZFS system, synchronization of the first cloudinstance and the second cloud instance based at least in part on one ormore remediation processes.
 8. A system comprising: one or moreprocessors communicatively coupled to memory, the one or more processorsto facilitate: receiving, from an application layer of a Z File System(ZFS) system and through a system call interface of an interface layerof the ZFS system, a file; creating, via one or more virtual devices ofa plurality of virtual devices at a virtual device layer of the ZFSsystem, data blocks that correspond to the file; causing, by a cloudinterface appliance, storage of cloud storage objects in a cloud objectstore, the cloud storage objects comprising data of the data blocks andcorresponding metadata of according to a tree hierarchy, wherein: afirst subset of the cloud storage objects comprises the data of the datablocks, the first subset corresponding to leaf nodes of the treehierarchy; a second subset of the cloud storage objects comprises thecorresponding metadata, the second subset corresponding to non-leafnodes of the tree hierarchy; and the corresponding metadata comprisesrespective checksums generated for each cloud storage object of thefirst subset of the cloud storage objects; receiving, from theapplication layer of the ZFS system, a request to perform a transactionwith respect to the file; translating, by a data management unit of atransactional object layer of the ZFS system, the request into an I/Orequest; based at least in part on the I/O request, performing, by thecloud interface appliance, a write operation to modify the treehierarchy at least in part by causing storage of an additional cloudstorage objects to result in a modified tree hierarchy, wherein: a thirdsubset of the additional cloud storage objects comprises additionaldata, the third subset corresponding to one or more additional leafnodes as modifications to the tree hierarchy; a fourth subset of theadditional cloud storage objects comprises additional metadatacorresponding to the additional data, the fourth subset corresponding toadditional non-leaf nodes of the tree hierarchy; and the additionalmetadata comprises additional respective checksums generated for eachcloud storage object of the third subset of the additional cloud storageobjects; based at least in part on a subsequent I/O request, performinga read operation with respect to the modified tree hierarchy, whereinthe read operation comprises receiving a version of at least one cloudstorage object from the cloud object store; and checking the version ofthe at least one cloud storage object with a reference checksum thatcorresponds to at least one of the respective checksums or at least oneof the additional respective checksums, where the reference checksum isstored locally with respect to the ZFS system and/or the cloud interfaceappliance or is received from the cloud object store.
 9. The system ofclaim 8, the one or more processors further to facilitate: receiving,from the application layer of the ZFS system, a subsequent request toperform a subsequent transaction with respect to the file; andtranslating, by the data management unit, the subsequent request intothe subsequent I/O request.
 10. The system of claim 9, wherein thechecking the version of the at least one cloud storage object with thereference checksum comprises: processing, by a storage pool allocator orthe cloud interface appliance, the version of the at least one cloudstorage object to generate a generated checksum; retrieving, by thestorage pool allocator or the cloud interface appliance, specificmetadata that is stored separately from the at least one cloud storageobject and that corresponds to a specific non-leaf node of the non-leafnodes or additional non-leaf nodes according to the modified treehierarchy, wherein the specific non-leaf node is a parent node withrespect to the at least one cloud storage object; identifying, by thestorage pool allocator or the cloud interface appliance, from thespecific metadata the reference checksum for the at least one cloudstorage object, the reference checksum being associated with a lastmodified version of the cloud storage object; and determining whetherthe generated checksum matches the reference checksum.
 11. The system ofclaim 10, wherein the performing, by the cloud interface appliance, thewrite operation to modify the tree hierarchy is performed according to acopy-on-write process.
 12. The system of claim 11, the one or moreprocessors further to facilitate: causing, via the cloud interfaceappliance or another cloud interface appliance, storage of the cloudstorage objects and the additional cloud storage objects in second cloudobject store so that a second cloud instance of the modified treehierarchy is stored in the second cloud object store, wherein themodified tree hierarchy stored in the cloud object store corresponds toa first cloud instance of the modified tree hierarchy; wherein thecausing storage of the cloud storage objects and the additional cloudstorage objects in the second cloud object store comprises independentgenerating additional checksums for each cloud storage object stored asleaf nodes in the second cloud object store.
 13. The system of claim 12,the one or more processors further to facilitate: after the storage ofthe cloud storage objects and the additional cloud storage objects inthe second cloud object store, identifying an unsynchronized state ofthe first cloud instance of the tree hierarchy with respect to thesecond cloud instance of the tree hierarchy based at least in part on atleast one checksum stored in at least one non-leaf node of the firstcloud instance and at least one checksum stored in at least one non-leafnode of the second cloud instance.
 14. The system of claim 13, the oneor more processors further to facilitate: in response to identifying theunsynchronized state, initiating, by the ZFS system, synchronization ofthe first cloud instance and the second cloud instance based at least inpart on one or more remediation processes.
 15. The system of claim 14,further comprising the ZFS system.
 16. One or more non-transitory,machine-readable media having machine-readable instructions thereonwhich, when executed by one or more processors, cause the one or moreprocessors to: receiving, from an application layer of a Z File System(ZFS) system and through a system call interface of an interface layerof the ZFS system, a file; creating, via one or more virtual devices ofa plurality of virtual devices at a virtual device layer of the ZFSsystem, data blocks that correspond to the file; causing, by a cloudinterface appliance, storage of cloud storage objects in a cloud objectstore, the cloud storage objects comprising data of the data blocks andcorresponding metadata of according to a tree hierarchy, wherein: afirst subset of the cloud storage objects comprises the data of the datablocks, the first subset corresponding to leaf nodes of the treehierarchy; a second subset of the cloud storage objects comprises thecorresponding metadata, the second subset corresponding to non-leafnodes of the tree hierarchy; and the corresponding metadata comprisesrespective checksums generated for each cloud storage object of thefirst subset of the cloud storage objects; receiving, from theapplication layer of the ZFS system, a request to perform a transactionwith respect to the file; translating, by a data management unit of atransactional object layer of the ZFS system, the request into an I/Orequest; based at least in part on the I/O request, performing, by thecloud interface appliance, a write operation to modify the treehierarchy at least in part by causing storage of an additional cloudstorage objects to result in a modified tree hierarchy, wherein: a thirdsubset of the additional cloud storage objects comprises additionaldata, the third subset corresponding to one or more additional leafnodes as modifications to the tree hierarchy; a fourth subset of theadditional cloud storage objects comprises additional metadatacorresponding to the additional data, the fourth subset corresponding toadditional non-leaf nodes of the tree hierarchy; and the additionalmetadata comprises additional respective checksums generated for eachcloud storage object of the third subset of the additional cloud storageobjects; based at least in part on a subsequent I/O request, performinga read operation with respect to the modified tree hierarchy, whereinthe read operation comprises receiving a version of at least one cloudstorage object from the cloud object store; and checking the version ofthe at least one cloud storage object with a reference checksum thatcorresponds to at least one of the respective checksums or at least oneof the additional respective checksums, where the reference checksum isstored locally with respect to the ZFS system and/or the cloud interfaceappliance or is received from the cloud object store.
 17. The one ormore non-transitory, machine-readable media of claim 16, the one or moreprocessors further to facilitate: receiving, from the application layerof the ZFS system, a subsequent request to perform a subsequenttransaction with respect to the file; and translating, by the datamanagement unit, the subsequent request into the subsequent I/O request.18. The one or more non-transitory, machine-readable media of claim 17,wherein the checking the version of the at least one cloud storageobject with the reference checksum comprises: processing, by a storagepool allocator or the cloud interface appliance, the version of the atleast one cloud storage object to generate a generated checksum;retrieving, by the storage pool allocator or the cloud interfaceappliance, specific metadata that is stored separately from the at leastone cloud storage object and that corresponds to a specific non-leafnode of the non-leaf nodes or additional non-leaf nodes according to themodified tree hierarchy, wherein the specific non-leaf node is a parentnode with respect to the at least one cloud storage object; identifying,by the storage pool allocator or the cloud interface appliance, from thespecific metadata the reference checksum for the at least one cloudstorage object, the reference checksum being associated with a lastmodified version of the cloud storage object; and determining whetherthe generated checksum matches the reference checksum.
 19. The one ormore non-transitory, machine-readable media of claim 18, wherein theperforming, by the cloud interface appliance, the write operation tomodify the tree hierarchy is performed according to a copy-on-writeprocess.
 20. The one or more non-transitory, machine-readable media ofclaim 19, the one or more processors further to facilitate: causing, viathe cloud interface appliance or another cloud interface appliance,storage of the cloud storage objects and the additional cloud storageobjects in second cloud object store so that a second cloud instance ofthe modified tree hierarchy is stored in the second cloud object store,wherein the modified tree hierarchy stored in the cloud object storecorresponds to a first cloud instance of the modified tree hierarchy;wherein the causing storage of the cloud storage objects and theadditional cloud storage objects in the second cloud object storecomprises independent generating additional checksums for each cloudstorage object stored as leaf nodes in the second cloud object store.